Sep 17 20:49:19.407032 ip-10-0-134-35 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 20:49:19.839440 ip-10-0-134-35 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:19.839440 ip-10-0-134-35 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 20:49:19.839440 ip-10-0-134-35 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:19.839440 ip-10-0-134-35 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 20:49:19.839440 ip-10-0-134-35 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:19.840397 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.840314    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 20:49:19.846731 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846716    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:19.846731 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846733    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846749    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846754    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846758    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846761    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846764    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846766    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846769    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846771    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846774    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846776    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846779    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846789    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846792    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846795    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846797    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846800    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846802    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846805    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846808    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:19.846824 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846810    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846813    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846816    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846819    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846821    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846823    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846826    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846829    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846832    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846834    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846837    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846840    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846842    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846845    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846847    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846850    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846852    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846854    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846857    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846860    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:19.847292 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846862    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846866    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846868    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846872    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846876    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846878    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846881    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846883    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846886    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846888    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846891    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846893    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846895    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846898    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846901    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846904    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846906    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846909    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846913    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:19.847795 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846915    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846918    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846920    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846922    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846924    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846927    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846929    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846932    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846935    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846937    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846940    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846942    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846945    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846947    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846950    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846953    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846956    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846958    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846960    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846963    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:19.848286 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846966    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846968    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846970    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846973    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846976    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846978    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846981    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846984    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846986    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846989    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846991    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846994    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846996    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.846999    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847001    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847003    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847006    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847008    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847011    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847687    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:19.848775 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847693    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847696    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847699    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847701    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847704    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847707    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847709    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847713    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847716    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847718    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847721    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847724    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847726    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847729    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847731    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847749    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847752    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847755    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847758    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847760    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:19.849253 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847763    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847766    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847769    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847771    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847774    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847777    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847780    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847783    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847785    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847788    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847790    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847793    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847796    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847799    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847801    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847804    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847807    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847809    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847812    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847814    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:19.849755 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847817    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847820    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847823    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847825    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847828    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847831    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847833    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847837    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847840    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847845    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847850    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847853    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847856    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847859    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847862    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847865    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847868    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847871    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847873    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:19.850244 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847876    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847878    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847881    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847883    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847886    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847888    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847891    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847894    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847897    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847900    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847903    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847906    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847909    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847912    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847914    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847919    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847922    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847924    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847927    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847929    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:19.850709 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847932    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847934    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847937    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847941    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847944    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847946    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847949    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847951    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847954    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847956    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847958    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847961    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847964    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847966    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847968    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847971    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847973    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847975    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.847978    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848341    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:19.851217 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848345    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848348    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848351    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848353    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848355    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848358    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848360    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848364    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848367    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848369    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848372    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848374    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848377    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848380    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848382    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848385    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848388    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848390    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848393    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848395    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:19.851706 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848398    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848400    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848403    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848405    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848409    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848412    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848414    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848418    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848422    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848425    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848427    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848430    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848433    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848436    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848439    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848441    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848443    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848446    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:19.852204 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848450    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848454    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848458    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848461    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848464    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848466    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848469    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848472    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848475    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848478    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848480    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848483    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848486    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848488    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848491    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848494    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848496    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848498    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848501    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848503    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:19.852637 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848506    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848508    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848511    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848513    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848515    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848518    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848520    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848523    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848525    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848527    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848530    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848532    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848534    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848537    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848540    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848543    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848545    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848547    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848550    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848553    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:19.853278 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848555    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848557    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848560    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848562    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848565    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848568    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848570    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848573    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848575    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848578    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848580    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848582    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848585    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848587    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848590    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848592    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848595    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848597    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848600    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.848602    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:19.853796 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848671    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848711    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848719    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848723    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848727    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848732    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848751    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848755    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848759    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848762    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848769    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848773    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848776    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848779    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848782    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848785    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848788    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848791    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848798    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848802    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848805    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848808    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848811    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848814    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 20:49:19.854279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848817    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848820    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848823    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848827    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848829    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848832    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848835    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848838    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848842    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848845    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848848    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848851    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848854    2577 flags.go:64] FLAG: --enable-server="true"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848857    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848861    2577 flags.go:64] FLAG: --event-burst="100"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848864    2577 flags.go:64] FLAG: --event-qps="50"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848867    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848870    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848875    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848879    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848881    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848884    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848887    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848890    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848893    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 20:49:19.854869 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848896    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848899    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848902    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848904    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848907    2577 flags.go:64] FLAG: --feature-gates=""
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848910    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848914    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848916    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848919    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848922    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848925    2577 flags.go:64] FLAG: --help="false"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848928    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-35.ec2.internal"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848931    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848933    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848936    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848939    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848942    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848945    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848948    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848951    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848954    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848957    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848960    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848963    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 17 20:49:19.855491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848965    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848970    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848973    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848977    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848980    2577 flags.go:64] FLAG: --lock-file=""
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848983    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848985    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848988    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848993    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848996    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.848999    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849002    2577 flags.go:64] FLAG: --logging-format="text"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849004    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849007    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849010    2577 flags.go:64] FLAG: --manifest-url=""
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849013    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849017    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849020    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849024    2577 flags.go:64] FLAG: --max-pods="110"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849027    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849030    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849033    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849036    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849039    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849041    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 20:49:19.856095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849044    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849052    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849055    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849058    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849061    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849064    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849069    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849072    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849075    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849079    2577 flags.go:64] FLAG: --port="10250"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849082    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849085    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d6749cda8c40ab6"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849088    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849091    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849094    2577 flags.go:64] FLAG: --register-node="true"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849097    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849100    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849103    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849106    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849108    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849112    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849115    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849118    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849121    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849123    2577 flags.go:64] FLAG: --runonce="false"
Sep 17 20:49:19.856756 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849126    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849129    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849132    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849135    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849138    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849141    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849144    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849147    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849150    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849153    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849156    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849159    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849162    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849165    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849167    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849173    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849177    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849180    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849184    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849187    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849189    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849192    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849195    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849198    2577 flags.go:64] FLAG: --v="2"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849202    2577 flags.go:64] FLAG: --version="false"
Sep 17 20:49:19.857385 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849206    2577 flags.go:64] FLAG: --vmodule=""
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849210    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.849213    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849343    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849347    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849350    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849353    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849359    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849361    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849365    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849368    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849370    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849373    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849375    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849378    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849380    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849383    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849387    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849390    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849393    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:19.858009 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849396    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849399    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849402    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849406    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849409    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849412    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849415    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849418    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849421    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849423    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849426    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849429    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849432    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849434    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849437    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849439    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849441    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849444    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849446    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849450    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:19.858501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849454    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849457    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849459    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849462    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849464    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849467    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849469    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849472    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849474    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849477    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849479    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849481    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849484    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849486    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849489    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849493    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849496    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849498    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849501    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849504    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:19.859015 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849507    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849509    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849511    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849514    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849516    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849519    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849521    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849524    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849526    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849528    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849531    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849534    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849537    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849539    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849541    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849544    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849547    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849549    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849552    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849554    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:19.859501 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849557    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849559    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849561    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849564    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849566    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849569    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849571    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849575    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849577    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849580    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849582    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849585    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849588    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849591    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849593    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849597    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849600    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849603    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849606    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:19.860060 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849609    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849612    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.849615    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.850331    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.858124    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.858140    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.858163    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.858173    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858229    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858234    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858237    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858240    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858243    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858246    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:19.860518 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858248    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858251    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858253    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858256    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858260    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858264    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858267    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858270    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858273    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858276    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858279    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858281    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858284    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858286    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858289    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858292    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858294    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858297    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858299    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858302    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:19.860898 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858304    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858306    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858309    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858313    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858316    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858318    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858321    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858324    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858326    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858329    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858331    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858334    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858336    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858348    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858350    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858353    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858356    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858359    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858361    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858363    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:19.861389 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858366    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858368    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858371    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858375    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858378    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858381    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858384    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858386    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858389    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858391    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858394    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858396    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858399    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858401    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858404    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858407    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858411    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858413    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858416    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:19.861966 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858418    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858421    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858424    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858427    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858429    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858432    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858435    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858437    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858440    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858442    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858444    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858447    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858451    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858454    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858457    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858460    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858462    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858465    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858467    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:19.862418 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858469    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858472    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858474    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858477    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858479    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858481    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858484    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858486    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858489    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858492    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858494    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858497    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858500    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858502    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858505    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:19.862906 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.858511    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858621    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858626    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858629    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858632    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858635    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858638    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858640    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858643    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858645    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858648    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858650    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858653    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858655    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858658    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858660    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858663    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858665    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858668    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858670    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:19.863277 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858673    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858675    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858677    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858680    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858682    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858686    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858688    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858692    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858695    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858698    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858701    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858703    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858706    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858708    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858711    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858714    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858717    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858720    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858722    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:19.863776 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858725    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858727    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858729    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858732    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858753    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858756    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858758    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858761    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858764    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858767    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858769    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858772    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858775    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858777    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858780    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858782    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858785    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858787    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858791    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858793    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:19.864243 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858796    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858798    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858801    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858805    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858807    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858810    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858813    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858816    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858820    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858823    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858825    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858829    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858831    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858834    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858837    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858839    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858842    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858846    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858849    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:19.864723 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858851    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858854    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858857    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858860    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858862    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858865    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858867    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858870    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858873    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858876    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858878    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858881    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858884    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858886    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858889    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858892    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858894    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858897    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858900    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:19.865196 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858902    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858905    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:19.858908    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.858913    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.859025    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.861393    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.862511    2577 server.go:1019] "Starting client certificate rotation"
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.862616    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.862655    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:19.865692 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.865288    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:19.870156 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.870128    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:19.885060 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.885034    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 17 20:49:19.886787 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.886770    2577 log.go:25] "Validated CRI v1 image API"
Sep 17 20:49:19.888463 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.888436    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 20:49:19.890992 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.890973    2577 fs.go:135] Filesystem UUIDs: map[6f1569ee-a254-4cd9-aa29-f8649877181c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d3f1820e-2845-4994-be65-3eedd178140c:/dev/nvme0n1p4]
Sep 17 20:49:19.891045 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.890993    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 20:49:19.895710 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.895565    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:19.896244 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.896129    2577 manager.go:216] Machine: {Timestamp:2026-09-17 20:49:19.89465956 +0000 UTC m=+0.378451861 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099975 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ab0e3421e816fb676405186b11eec SystemUUID:ec2ab0e3-421e-816f-b676-405186b11eec BootID:821e87d0-d78f-4da4-9b61-9b5dae0f4060 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:d8:fe:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:d8:fe:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:73:5b:e2:5d:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 20:49:19.896244 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.896242    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 20:49:19.896335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.896327    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 20:49:19.899688 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.899665    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 20:49:19.899862 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.899692    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 20:49:19.901227 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.901216    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 20:49:19.901265 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.901230    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 20:49:19.901265 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.901238    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:19.902056 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.902045    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:19.902133 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.902125    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 20:49:19.906895 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.906884    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:19.907015 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907005    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 20:49:19.907054 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907037    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 20:49:19.907054 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907046    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 20:49:19.907121 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907060    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 20:49:19.907121 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907071    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 17 20:49:19.907121 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907079    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 20:49:19.907465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907454    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:19.907507 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.907474    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:19.909814 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.909781    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 20:49:19.912873 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.912855    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 20:49:19.912942 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.912889    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 20:49:19.914251 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914239    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 20:49:19.914291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914255    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 20:49:19.914291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914261    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 20:49:19.914291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914266    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 20:49:19.914291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914275    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 20:49:19.914291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914281    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914294    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914299    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914309    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914315    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914326    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914331    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914362    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 20:49:19.914465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.914368    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 20:49:19.917781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.917763    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-35.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 20:49:19.917860 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:19.917823    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 20:49:19.917860 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:19.917836    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 20:49:19.918437 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.918420    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kpctv"
Sep 17 20:49:19.918518 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.918502    2577 server.go:1268] "Started kubelet"
Sep 17 20:49:19.918581 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.918535    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 20:49:19.918709 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.918641    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 20:49:19.918765 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.918750    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 20:49:19.919337 ip-10-0-134-35 systemd[1]: Started Kubernetes Kubelet.
Sep 17 20:49:19.923142 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.923123    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 20:49:19.925633 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.925615    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 20:49:19.926025 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:19.926003    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 20:49:19.926701 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.926687    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kpctv"
Sep 17 20:49:19.928022 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:19.926858    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-35.ec2.internal.18d636cde836c924  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-35.ec2.internal,UID:ip-10-0-134-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-35.ec2.internal,},FirstTimestamp:2026-09-17 20:49:19.91845098 +0000 UTC m=+0.402243263,LastTimestamp:2026-09-17 20:49:19.91845098 +0000 UTC m=+0.402243263,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-35.ec2.internal,}"
Sep 17 20:49:19.935072 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.935058    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 20:49:19.935142 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.935081    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:19.935198 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:19.935180    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:19.935805 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.935783    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 20:49:19.935805 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.935787    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 20:49:19.936022 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.935817    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 20:49:19.936022 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.935899    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 20:49:19.936022 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.935907    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 20:49:19.936444 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.936418    2577 factory.go:55] Registering systemd factory
Sep 17 20:49:19.936444 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.936446    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 17 20:49:19.936685 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.936671    2577 factory.go:153] Registering CRI-O factory
Sep 17 20:49:19.936762 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.936688    2577 factory.go:223] Registration of the crio container factory successfully
Sep 17 20:49:19.936823 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.936759    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 20:49:19.936823 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.936780    2577 factory.go:103] Registering Raw factory
Sep 17 20:49:19.936823 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.936796    2577 manager.go:1196] Started watching for new ooms in manager
Sep 17 20:49:19.937164 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.937150    2577 manager.go:349] Starting recovery of all containers
Sep 17 20:49:19.947060 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.947043    2577 manager.go:354] Recovery completed
Sep 17 20:49:19.947319 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.947301    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:19.949620 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:19.949603    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-35.ec2.internal\" not found" node="ip-10-0-134-35.ec2.internal"
Sep 17 20:49:19.951695 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.951683    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:19.954248 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.954234    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:19.954306 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.954262    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:19.954306 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.954275    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:19.954807 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.954794    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 20:49:19.954862 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.954807    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:19.954862 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.954808    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 20:49:19.954918 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.954878    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:19.955501 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.955488    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:19.955551 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.955513    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:19.955551 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.955532    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:19.956949 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.956938    2577 policy_none.go:49] "None policy: Start"
Sep 17 20:49:19.956990 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.956954    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 20:49:19.956990 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.956964    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 20:49:19.958919 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:19.958908    2577 policy_none.go:47] "Start"
Sep 17 20:49:20.010369 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.010349    2577 manager.go:337] "Starting Device Plugin manager"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.010410    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.010422    2577 server.go:87] "Starting device plugin registration server"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.010802    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.010815    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.010904    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.010978    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.010986    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.011541    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 20:49:20.020852 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.011594    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.045533    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 20:49:20.046874 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.046856    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 20:49:20.046947 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.046904    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 20:49:20.046947 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.046925    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 20:49:20.047021 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.046968    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 20:49:20.049087 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.049067    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:20.111114 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.111022    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.111976 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.111961    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.112035 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.111991    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.112035 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.112008    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.112035 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.112034    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.119962 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.119949    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.120021 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.119968    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-35.ec2.internal\": node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.131123 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.131101    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.147529 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.147505    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"]
Sep 17 20:49:20.147612 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.147578    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.148870 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.148854    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.148936 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.148885    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.148936 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.148895    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.151214 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151202    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.151332 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.151376 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151345    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.151906 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151886    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.151999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151917    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.151999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151931    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.151999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151891    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.151999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.151994    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.152180 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.152005    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.154289 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.154274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.154360 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.154301    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.155123 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.155107    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.155197 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.155139    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.155197 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.155150    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.180346 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.180326    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-35.ec2.internal\" not found" node="ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.184826 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.184810    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-35.ec2.internal\" not found" node="ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.231859 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.231840    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.238166 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.238147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8722866d1367aa26c31e0552f2a8a7af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal\" (UID: \"8722866d1367aa26c31e0552f2a8a7af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.238237 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.238176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8722866d1367aa26c31e0552f2a8a7af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal\" (UID: \"8722866d1367aa26c31e0552f2a8a7af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.238237 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.238216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43a2109c14934db412a36a19c6e1114f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-35.ec2.internal\" (UID: \"43a2109c14934db412a36a19c6e1114f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.332043 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.332021    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.338300 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.338280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8722866d1367aa26c31e0552f2a8a7af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal\" (UID: \"8722866d1367aa26c31e0552f2a8a7af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.338360 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.338313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8722866d1367aa26c31e0552f2a8a7af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal\" (UID: \"8722866d1367aa26c31e0552f2a8a7af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.338360 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.338329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43a2109c14934db412a36a19c6e1114f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-35.ec2.internal\" (UID: \"43a2109c14934db412a36a19c6e1114f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.338453 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.338371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43a2109c14934db412a36a19c6e1114f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-35.ec2.internal\" (UID: \"43a2109c14934db412a36a19c6e1114f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.338453 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.338389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8722866d1367aa26c31e0552f2a8a7af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal\" (UID: \"8722866d1367aa26c31e0552f2a8a7af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.338453 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.338407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8722866d1367aa26c31e0552f2a8a7af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal\" (UID: \"8722866d1367aa26c31e0552f2a8a7af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.432809 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.432706    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.483170 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.483140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.488586 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.488462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.533476 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.533452    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.634026 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.633992    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.734538 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.734476    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-35.ec2.internal\" not found"
Sep 17 20:49:20.820642 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.820616    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:20.835349 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.835326    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.859246 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.859224    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:20.860633 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.860614    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.862378 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.862364    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 20:49:20.862478 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.862463    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:20.862520 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.862506    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:20.862555 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.862508    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:20.862555 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.862526    2577 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a2ee0083f4cf34d82b92ec18c8cbfa6f-322a17d1696c41c7.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.134.35:48308->184.194.89.179:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:20.907685 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.907666    2577 apiserver.go:52] "Watching apiserver"
Sep 17 20:49:20.927806 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.927786    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 20:49:20.928118 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.928099    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-nzjvb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t","openshift-image-registry/node-ca-m6lqg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal","openshift-multus/multus-additional-cni-plugins-hfgtm","openshift-multus/network-metrics-daemon-4tl6x","openshift-network-diagnostics/network-check-target-h68z9","openshift-network-operator/iptables-alerter-gqsj2","kube-system/global-pull-secret-syncer-lsbm5","openshift-cluster-node-tuning-operator/tuned-v4g8x","openshift-dns/node-resolver-h5ltl","openshift-multus/multus-fv4xg","openshift-ovn-kubernetes/ovnkube-node-pdn6f"]
Sep 17 20:49:20.930585 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.930571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.932652 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.932627    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:44:21 +0000 UTC" deadline="2028-06-01 10:24:29.94917303 +0000 UTC"
Sep 17 20:49:20.932721 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.932652    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14941h35m9.016523899s"
Sep 17 20:49:20.932798 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.932785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.935023 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:20.935224 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935208    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:20.935323 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935307    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 20:49:20.935382 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 20:49:20.935382 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:20.935475 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935426    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c8k8s\""
Sep 17 20:49:20.935475 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935313    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:20.935583 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.935564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m2snm\""
Sep 17 20:49:20.937852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.937526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.939470 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.937943    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 20:49:20.940671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.940651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:20.940777 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.940728    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:20.941583 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49392876-888a-4002-a542-faa186562b69-serviceca\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:20.941635 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-modprobe-d\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.941635 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-device-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.941635 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.941781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-kubernetes\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.941781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-systemd\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.941781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cl4p\" (UniqueName: \"kubernetes.io/projected/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-kube-api-access-2cl4p\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.941781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-run\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.941781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-tuned\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-sys-fs\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-cnibin\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-sys\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rpn\" (UniqueName: \"kubernetes.io/projected/49392876-888a-4002-a542-faa186562b69-kube-api-access-w6rpn\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-cni-binary-copy\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49392876-888a-4002-a542-faa186562b69-host\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:20.941954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysconfig\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysctl-conf\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.941994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-socket-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsx4w\" (UniqueName: \"kubernetes.io/projected/580f0aac-e36f-422d-b6dd-959e3b57d61f-kube-api-access-wsx4w\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-lib-modules\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-host\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.942226 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j624g\" (UniqueName: \"kubernetes.io/projected/72503c7a-e043-410b-b606-d4673b7c157c-kube-api-access-j624g\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.942490 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-system-cni-dir\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.942490 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysctl-d\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.942490 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-var-lib-kubelet\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.942490 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-tmp\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:20.942490 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-registration-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:20.942490 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.942363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:20.943597 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.943579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:20.943656 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.943641    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:20.945392 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945376    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 20:49:20.945392 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 20:49:20.945525 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945391    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 20:49:20.945525 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945446    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lw42v\""
Sep 17 20:49:20.945525 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 20:49:20.945525 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 20:49:20.945704 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 20:49:20.945833 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:20.945917 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.945903    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 20:49:20.946125 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.946094    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-29hz2\""
Sep 17 20:49:20.946198 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.946183    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 20:49:20.947877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.947863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:20.947960 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:20.947938    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:20.950556 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.950543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:20.952874 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.952858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:20.953340 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.953326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 20:49:20.953416 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.953399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wrntt\""
Sep 17 20:49:20.954549 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.954535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 20:49:20.954983 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.954968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:20.956567 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.956549    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 20:49:20.956655 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.956636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7jxcv\""
Sep 17 20:49:20.957885 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.957870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:20.959331 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.959312    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:20.959416 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.959334    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 20:49:20.959416 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.959344    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72xgz\""
Sep 17 20:49:20.959416 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.959312    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:20.959727 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.959711    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 20:49:20.963785 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.963770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 20:49:20.963984 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.963920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5qh6s\""
Sep 17 20:49:20.967403 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.967388    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 20:49:20.972687 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.972667    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 20:49:20.972792 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.972770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 20:49:20.972850 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.972814    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5xqt6\""
Sep 17 20:49:20.974253 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.974237    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 20:49:20.976684 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.976670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 20:49:20.978003 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.977989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 20:49:20.981188 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.981087    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:20.991575 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:20.991535    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:21.033874 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.033849    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fhvnb"
Sep 17 20:49:21.037811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.037790    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 20:49:21.043126 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-lib-modules\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.043230 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-host\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.043230 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j624g\" (UniqueName: \"kubernetes.io/projected/72503c7a-e043-410b-b606-d4673b7c157c-kube-api-access-j624g\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.043230 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-system-cni-dir\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.043230 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-system-cni-dir\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hcp\" (UniqueName: \"kubernetes.io/projected/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-kube-api-access-k4hcp\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-host\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-etc-kubernetes\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-lib-modules\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovnkube-config\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-tmp\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.043402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28sn8\" (UniqueName: \"kubernetes.io/projected/5afbfc91-924c-42d4-8e64-fcaac85986e4-kube-api-access-28sn8\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-cni-multus\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-var-lib-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c4051-4a21-4899-8871-eb7c00e693a2-tmp-dir\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-ovn\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-node-log\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-cni-bin\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49392876-888a-4002-a542-faa186562b69-serviceca\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-modprobe-d\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-device-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.043671 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-system-cni-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-slash\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-etc-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-env-overrides\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-modprobe-d\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-systemd\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cl4p\" (UniqueName: \"kubernetes.io/projected/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-kube-api-access-2cl4p\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72932197-8c1b-4d17-9e63-7cc9b1de8316-agent-certs\") pod \"konnectivity-agent-nzjvb\" (UID: \"72932197-8c1b-4d17-9e63-7cc9b1de8316\") " pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-device-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9nk\" (UniqueName: \"kubernetes.io/projected/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-kube-api-access-6w9nk\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-run\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-tuned\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-sys-fs\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-cni-binary-copy\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.043982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.044032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044009    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49392876-888a-4002-a542-faa186562b69-serviceca\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-run\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-cni-bin\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-systemd\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-sys-fs\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-daemon-config\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-netns\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rpn\" (UniqueName: \"kubernetes.io/projected/49392876-888a-4002-a542-faa186562b69-kube-api-access-w6rpn\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-cni-binary-copy\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbxg\" (UniqueName: \"kubernetes.io/projected/141c4051-4a21-4899-8871-eb7c00e693a2-kube-api-access-gmbxg\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-hostroot\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysconfig\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysctl-conf\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.044773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-conf-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-systemd\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-cnibin\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-kubelet\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-log-socket\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysctl-d\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-var-lib-kubelet\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5afbfc91-924c-42d4-8e64-fcaac85986e4-iptables-alerter-script\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72932197-8c1b-4d17-9e63-7cc9b1de8316-konnectivity-ca\") pod \"konnectivity-agent-nzjvb\" (UID: \"72932197-8c1b-4d17-9e63-7cc9b1de8316\") " pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-cni-binary-copy\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-cni-netd\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-registration-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-cni-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.045564 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysconfig\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/580f0aac-e36f-422d-b6dd-959e3b57d61f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.044970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-multus-certs\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-registration-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysctl-conf\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-kubernetes\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-cnibin\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843bd172-4a4b-4285-8810-a1edee197936-dbus\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-k8s-cni-cncf-io\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-run-netns\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovn-node-metrics-cert\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-var-lib-kubelet\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-socket-dir-parent\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-sys\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-sysctl-d\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580f0aac-e36f-422d-b6dd-959e3b57d61f-cnibin\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843bd172-4a4b-4285-8810-a1edee197936-kubelet-config\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/141c4051-4a21-4899-8871-eb7c00e693a2-hosts-file\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-kubernetes\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-kubelet\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-systemd-units\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-sys\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovnkube-script-lib\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5afbfc91-924c-42d4-8e64-fcaac85986e4-host-slash\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49392876-888a-4002-a542-faa186562b69-host\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49392876-888a-4002-a542-faa186562b69-host\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-socket-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx4w\" (UniqueName: \"kubernetes.io/projected/580f0aac-e36f-422d-b6dd-959e3b57d61f-kube-api-access-wsx4w\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dbj\" (UniqueName: \"kubernetes.io/projected/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-kube-api-access-v2dbj\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.045844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72503c7a-e043-410b-b606-d4673b7c157c-socket-dir\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.046910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.046345    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fhvnb"
Sep 17 20:49:21.047882 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.047861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-tmp\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.048297 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.048280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-etc-tuned\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.069335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.069176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx4w\" (UniqueName: \"kubernetes.io/projected/580f0aac-e36f-422d-b6dd-959e3b57d61f-kube-api-access-wsx4w\") pod \"multus-additional-cni-plugins-hfgtm\" (UID: \"580f0aac-e36f-422d-b6dd-959e3b57d61f\") " pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.069613 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.069184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rpn\" (UniqueName: \"kubernetes.io/projected/49392876-888a-4002-a542-faa186562b69-kube-api-access-w6rpn\") pod \"node-ca-m6lqg\" (UID: \"49392876-888a-4002-a542-faa186562b69\") " pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:21.071899 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.071880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cl4p\" (UniqueName: \"kubernetes.io/projected/cb7e92bb-0e76-4a92-b79a-a74c6d4b751a-kube-api-access-2cl4p\") pod \"tuned-v4g8x\" (UID: \"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.075933 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.075918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j624g\" (UniqueName: \"kubernetes.io/projected/72503c7a-e043-410b-b606-d4673b7c157c-kube-api-access-j624g\") pod \"aws-ebs-csi-driver-node-jf48t\" (UID: \"72503c7a-e043-410b-b606-d4673b7c157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.146146 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-cni-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.146146 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-multus-certs\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843bd172-4a4b-4285-8810-a1edee197936-dbus\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-k8s-cni-cncf-io\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-run-netns\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovn-node-metrics-cert\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-socket-dir-parent\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-k8s-cni-cncf-io\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-run-netns\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843bd172-4a4b-4285-8810-a1edee197936-kubelet-config\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-socket-dir-parent\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-cni-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843bd172-4a4b-4285-8810-a1edee197936-dbus\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/141c4051-4a21-4899-8871-eb7c00e693a2-hosts-file\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.146354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-multus-certs\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843bd172-4a4b-4285-8810-a1edee197936-kubelet-config\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-kubelet\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-kubelet\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-systemd-units\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovnkube-script-lib\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/141c4051-4a21-4899-8871-eb7c00e693a2-hosts-file\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-systemd-units\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5afbfc91-924c-42d4-8e64-fcaac85986e4-host-slash\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dbj\" (UniqueName: \"kubernetes.io/projected/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-kube-api-access-v2dbj\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hcp\" (UniqueName: \"kubernetes.io/projected/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-kube-api-access-k4hcp\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5afbfc91-924c-42d4-8e64-fcaac85986e4-host-slash\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.146542    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-etc-kubernetes\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.146666    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:21.646627092 +0000 UTC m=+2.130419365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-etc-kubernetes\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147032 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.146676    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovnkube-config\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.146715    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret podName:843bd172-4a4b-4285-8810-a1edee197936 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:21.646703564 +0000 UTC m=+2.130495836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret") pod "global-pull-secret-syncer-lsbm5" (UID: "843bd172-4a4b-4285-8810-a1edee197936") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28sn8\" (UniqueName: \"kubernetes.io/projected/5afbfc91-924c-42d4-8e64-fcaac85986e4-kube-api-access-28sn8\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-cni-multus\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-var-lib-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c4051-4a21-4899-8871-eb7c00e693a2-tmp-dir\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-ovn\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-node-log\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-cni-bin\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-var-lib-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-system-cni-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-cni-multus\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-slash\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-etc-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-env-overrides\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.146995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-node-log\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.147866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72932197-8c1b-4d17-9e63-7cc9b1de8316-agent-certs\") pod \"konnectivity-agent-nzjvb\" (UID: \"72932197-8c1b-4d17-9e63-7cc9b1de8316\") " pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9nk\" (UniqueName: \"kubernetes.io/projected/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-kube-api-access-6w9nk\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovnkube-script-lib\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-ovn\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-cni-binary-copy\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-cni-bin\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-cni-bin\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-var-lib-cni-bin\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-etc-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-slash\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovnkube-config\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-system-cni-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-daemon-config\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-netns\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c4051-4a21-4899-8871-eb7c00e693a2-tmp-dir\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-host-run-netns\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbxg\" (UniqueName: \"kubernetes.io/projected/141c4051-4a21-4899-8871-eb7c00e693a2-kube-api-access-gmbxg\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-hostroot\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-conf-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-systemd\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-hostroot\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-cnibin\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-kubelet\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-conf-dir\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-log-socket\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5afbfc91-924c-42d4-8e64-fcaac85986e4-iptables-alerter-script\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-kubelet\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72932197-8c1b-4d17-9e63-7cc9b1de8316-konnectivity-ca\") pod \"konnectivity-agent-nzjvb\" (UID: \"72932197-8c1b-4d17-9e63-7cc9b1de8316\") " pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-systemd\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-cni-netd\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-cnibin\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.148980 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-env-overrides\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-run-openvswitch\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-host-cni-netd\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-log-socket\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.147975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-cni-binary-copy\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.148009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-multus-daemon-config\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.148277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5afbfc91-924c-42d4-8e64-fcaac85986e4-iptables-alerter-script\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.148355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72932197-8c1b-4d17-9e63-7cc9b1de8316-konnectivity-ca\") pod \"konnectivity-agent-nzjvb\" (UID: \"72932197-8c1b-4d17-9e63-7cc9b1de8316\") " pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.148827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-ovn-node-metrics-cert\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.149493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.149469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72932197-8c1b-4d17-9e63-7cc9b1de8316-agent-certs\") pod \"konnectivity-agent-nzjvb\" (UID: \"72932197-8c1b-4d17-9e63-7cc9b1de8316\") " pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:21.162314 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.162290    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:21.162314 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.162314    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:21.162462 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.162327    2577 projected.go:196] Error preparing data for projected volume kube-api-access-667sr for pod openshift-network-diagnostics/network-check-target-h68z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:21.162462 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:21.162392    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr podName:a35a3ee0-24f0-473c-a046-2ef32cafb6a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:21.662375747 +0000 UTC m=+2.146168021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-667sr" (UniqueName: "kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr") pod "network-check-target-h68z9" (UID: "a35a3ee0-24f0-473c-a046-2ef32cafb6a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:21.170482 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.170457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dbj\" (UniqueName: \"kubernetes.io/projected/8ed1c2f5-88fd-453f-9b58-82b7e81fc60b-kube-api-access-v2dbj\") pod \"multus-fv4xg\" (UID: \"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b\") " pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.170578 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.170522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbxg\" (UniqueName: \"kubernetes.io/projected/141c4051-4a21-4899-8871-eb7c00e693a2-kube-api-access-gmbxg\") pod \"node-resolver-h5ltl\" (UID: \"141c4051-4a21-4899-8871-eb7c00e693a2\") " pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.171849 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.171822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hcp\" (UniqueName: \"kubernetes.io/projected/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-kube-api-access-k4hcp\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:21.172019 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.171996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9nk\" (UniqueName: \"kubernetes.io/projected/ff7b3b81-da9c-4d8c-8946-6d28cf2087e4-kube-api-access-6w9nk\") pod \"ovnkube-node-pdn6f\" (UID: \"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.172019 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.172011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28sn8\" (UniqueName: \"kubernetes.io/projected/5afbfc91-924c-42d4-8e64-fcaac85986e4-kube-api-access-28sn8\") pod \"iptables-alerter-gqsj2\" (UID: \"5afbfc91-924c-42d4-8e64-fcaac85986e4\") " pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.183108 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.183067    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8722866d1367aa26c31e0552f2a8a7af.slice/crio-d8bff128811246809c022e51aab15f3d2629b31a2c2c2364910296f59237f84d WatchSource:0}: Error finding container d8bff128811246809c022e51aab15f3d2629b31a2c2c2364910296f59237f84d: Status 404 returned error can't find the container with id d8bff128811246809c022e51aab15f3d2629b31a2c2c2364910296f59237f84d
Sep 17 20:49:21.183372 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.183353    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a2109c14934db412a36a19c6e1114f.slice/crio-499ed3907c20f2e2d0da85b563f8911dce76d0608378963b6b26a608d08e86e0 WatchSource:0}: Error finding container 499ed3907c20f2e2d0da85b563f8911dce76d0608378963b6b26a608d08e86e0: Status 404 returned error can't find the container with id 499ed3907c20f2e2d0da85b563f8911dce76d0608378963b6b26a608d08e86e0
Sep 17 20:49:21.188199 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.188185    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:49:21.263621 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.263571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4g8x"
Sep 17 20:49:21.280613 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.280593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t"
Sep 17 20:49:21.288850 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.288832    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72503c7a_e043_410b_b606_d4673b7c157c.slice/crio-8c65066415d4f177d5011a43a0e36dc76c1bcf29590283fa5421ebc3ea2e2ea1 WatchSource:0}: Error finding container 8c65066415d4f177d5011a43a0e36dc76c1bcf29590283fa5421ebc3ea2e2ea1: Status 404 returned error can't find the container with id 8c65066415d4f177d5011a43a0e36dc76c1bcf29590283fa5421ebc3ea2e2ea1
Sep 17 20:49:21.303152 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.303135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m6lqg"
Sep 17 20:49:21.311439 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.311420    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49392876_888a_4002_a542_faa186562b69.slice/crio-b713b8c07e1190507047063bded4f5de10e2209576a974fc5896d72337546c75 WatchSource:0}: Error finding container b713b8c07e1190507047063bded4f5de10e2209576a974fc5896d72337546c75: Status 404 returned error can't find the container with id b713b8c07e1190507047063bded4f5de10e2209576a974fc5896d72337546c75
Sep 17 20:49:21.316469 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.316454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hfgtm"
Sep 17 20:49:21.324248 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.324231    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580f0aac_e36f_422d_b6dd_959e3b57d61f.slice/crio-c38cdc20b5b91bb909bc631786681aa1e0d079cbf3384bb0169f6517175c831a WatchSource:0}: Error finding container c38cdc20b5b91bb909bc631786681aa1e0d079cbf3384bb0169f6517175c831a: Status 404 returned error can't find the container with id c38cdc20b5b91bb909bc631786681aa1e0d079cbf3384bb0169f6517175c831a
Sep 17 20:49:21.331957 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.331939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqsj2"
Sep 17 20:49:21.339466 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.339450    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afbfc91_924c_42d4_8e64_fcaac85986e4.slice/crio-dcb59ff9deaf2a5b9ee70cf34a395a6505c2dab04768d0ba0d3ee36a05be79c4 WatchSource:0}: Error finding container dcb59ff9deaf2a5b9ee70cf34a395a6505c2dab04768d0ba0d3ee36a05be79c4: Status 404 returned error can't find the container with id dcb59ff9deaf2a5b9ee70cf34a395a6505c2dab04768d0ba0d3ee36a05be79c4
Sep 17 20:49:21.345672 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.345657    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:21.353055 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.353037    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72932197_8c1b_4d17_9e63_7cc9b1de8316.slice/crio-cdb1fd35e7b2fb455cb29174c9be2b09ee90fd8c71375fd840920985a0d8e641 WatchSource:0}: Error finding container cdb1fd35e7b2fb455cb29174c9be2b09ee90fd8c71375fd840920985a0d8e641: Status 404 returned error can't find the container with id cdb1fd35e7b2fb455cb29174c9be2b09ee90fd8c71375fd840920985a0d8e641
Sep 17 20:49:21.356027 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.356012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h5ltl"
Sep 17 20:49:21.363502 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.363485    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141c4051_4a21_4899_8871_eb7c00e693a2.slice/crio-6410d5259eee9fe8aa761a3c90aa50961c0611bdbe0b6b24bba7bbb62be80e58 WatchSource:0}: Error finding container 6410d5259eee9fe8aa761a3c90aa50961c0611bdbe0b6b24bba7bbb62be80e58: Status 404 returned error can't find the container with id 6410d5259eee9fe8aa761a3c90aa50961c0611bdbe0b6b24bba7bbb62be80e58
Sep 17 20:49:21.378193 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.378178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fv4xg"
Sep 17 20:49:21.381936 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:21.381921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:21.386227 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.386208    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed1c2f5_88fd_453f_9b58_82b7e81fc60b.slice/crio-31a873b2e2846c87cb812c3b1068e6b29dcb5ab950598775f36a80bcf3eea72f WatchSource:0}: Error finding container 31a873b2e2846c87cb812c3b1068e6b29dcb5ab950598775f36a80bcf3eea72f: Status 404 returned error can't find the container with id 31a873b2e2846c87cb812c3b1068e6b29dcb5ab950598775f36a80bcf3eea72f
Sep 17 20:49:21.390613 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:21.390595    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7b3b81_da9c_4d8c_8946_6d28cf2087e4.slice/crio-3ac063068fadaa2ba7b61602855644260c4a2cc894d343dd053ccddb31e71d7a WatchSource:0}: Error finding container 3ac063068fadaa2ba7b61602855644260c4a2cc894d343dd053ccddb31e71d7a: Status 404 returned error can't find the container with id 3ac063068fadaa2ba7b61602855644260c4a2cc894d343dd053ccddb31e71d7a
Sep 17 20:49:23.317501 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.317444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:23.317501 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.317494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:23.317769 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.317609    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:23.317769 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.317618    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:23.317769 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.317667    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.317652665 +0000 UTC m=+3.136944630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:23.317769 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.317688    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret podName:843bd172-4a4b-4285-8810-a1edee197936 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.317677036 +0000 UTC m=+3.136968981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret") pod "global-pull-secret-syncer-lsbm5" (UID: "843bd172-4a4b-4285-8810-a1edee197936") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:23.418570 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.418538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:23.418729 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.418701    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:23.418787 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.418734    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:23.418787 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.418744    2577 projected.go:196] Error preparing data for projected volume kube-api-access-667sr for pod openshift-network-diagnostics/network-check-target-h68z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:23.418871 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:23.418797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr podName:a35a3ee0-24f0-473c-a046-2ef32cafb6a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.4187836 +0000 UTC m=+3.238075554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-667sr" (UniqueName: "kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr") pod "network-check-target-h68z9" (UID: "a35a3ee0-24f0-473c-a046-2ef32cafb6a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:23.683490 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.683391    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:23.715662 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.715520    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:22 +0000 UTC" deadline="2028-06-02 18:37:48.765015346 +0000 UTC"
Sep 17 20:49:23.715662 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.715553    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14973h48m25.049467103s"
Sep 17 20:49:23.735518 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.735386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"3ac063068fadaa2ba7b61602855644260c4a2cc894d343dd053ccddb31e71d7a"}
Sep 17 20:49:23.746192 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.746122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fv4xg" event={"ID":"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b","Type":"ContainerStarted","Data":"31a873b2e2846c87cb812c3b1068e6b29dcb5ab950598775f36a80bcf3eea72f"}
Sep 17 20:49:23.755615 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.755519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nzjvb" event={"ID":"72932197-8c1b-4d17-9e63-7cc9b1de8316","Type":"ContainerStarted","Data":"cdb1fd35e7b2fb455cb29174c9be2b09ee90fd8c71375fd840920985a0d8e641"}
Sep 17 20:49:23.770097 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.770064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m6lqg" event={"ID":"49392876-888a-4002-a542-faa186562b69","Type":"ContainerStarted","Data":"b713b8c07e1190507047063bded4f5de10e2209576a974fc5896d72337546c75"}
Sep 17 20:49:23.777698 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.777666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t" event={"ID":"72503c7a-e043-410b-b606-d4673b7c157c","Type":"ContainerStarted","Data":"8c65066415d4f177d5011a43a0e36dc76c1bcf29590283fa5421ebc3ea2e2ea1"}
Sep 17 20:49:23.787194 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.787168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal" event={"ID":"43a2109c14934db412a36a19c6e1114f","Type":"ContainerStarted","Data":"499ed3907c20f2e2d0da85b563f8911dce76d0608378963b6b26a608d08e86e0"}
Sep 17 20:49:23.792588 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.792465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal" event={"ID":"8722866d1367aa26c31e0552f2a8a7af","Type":"ContainerStarted","Data":"d8bff128811246809c022e51aab15f3d2629b31a2c2c2364910296f59237f84d"}
Sep 17 20:49:23.795641 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.795615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h5ltl" event={"ID":"141c4051-4a21-4899-8871-eb7c00e693a2","Type":"ContainerStarted","Data":"6410d5259eee9fe8aa761a3c90aa50961c0611bdbe0b6b24bba7bbb62be80e58"}
Sep 17 20:49:23.809705 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.809608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqsj2" event={"ID":"5afbfc91-924c-42d4-8e64-fcaac85986e4","Type":"ContainerStarted","Data":"dcb59ff9deaf2a5b9ee70cf34a395a6505c2dab04768d0ba0d3ee36a05be79c4"}
Sep 17 20:49:23.822824 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.822795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerStarted","Data":"c38cdc20b5b91bb909bc631786681aa1e0d079cbf3384bb0169f6517175c831a"}
Sep 17 20:49:23.851583 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.851549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4g8x" event={"ID":"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a","Type":"ContainerStarted","Data":"2c44f437b3ec8ada36a107e41bc6f9048584ee0687588034cd445af713763e7c"}
Sep 17 20:49:23.921256 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:23.921217    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:24.326339 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.326284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:24.326491 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.326370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:24.326561 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.326541    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.326667 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.326645    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret podName:843bd172-4a4b-4285-8810-a1edee197936 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:26.326623708 +0000 UTC m=+5.145915659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret") pod "global-pull-secret-syncer-lsbm5" (UID: "843bd172-4a4b-4285-8810-a1edee197936") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.327143 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.327123    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:24.327225 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.327183    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:26.327167409 +0000 UTC m=+5.146459356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:24.426892 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.426860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:24.427083 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.427061    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:24.427083 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.427081    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:24.427204 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.427091    2577 projected.go:196] Error preparing data for projected volume kube-api-access-667sr for pod openshift-network-diagnostics/network-check-target-h68z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:24.427204 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.427139    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr podName:a35a3ee0-24f0-473c-a046-2ef32cafb6a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:26.427126809 +0000 UTC m=+5.246418753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-667sr" (UniqueName: "kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr") pod "network-check-target-h68z9" (UID: "a35a3ee0-24f0-473c-a046-2ef32cafb6a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:24.712337 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.712257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:24.712783 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.712390    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:24.712855 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.712787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:24.713106 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.712894    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:24.713106 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.712972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:24.713106 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:24.713048    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:24.716335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.716304    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:22 +0000 UTC" deadline="2028-06-06 20:35:48.984007202 +0000 UTC"
Sep 17 20:49:24.716335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:24.716334    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15071h46m24.267678525s"
Sep 17 20:49:25.399267 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:25.398937    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:26.344072 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:26.343253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:26.344072 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:26.343311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:26.344072 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.343481    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:26.344072 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.343542    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret podName:843bd172-4a4b-4285-8810-a1edee197936 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:30.343523095 +0000 UTC m=+9.162815053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret") pod "global-pull-secret-syncer-lsbm5" (UID: "843bd172-4a4b-4285-8810-a1edee197936") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:26.344072 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.343960    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:26.344072 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.344013    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:30.343996052 +0000 UTC m=+9.163288002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:26.444297 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:26.443634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:26.444297 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.443834    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:26.444297 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.443854    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:26.444297 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.443866    2577 projected.go:196] Error preparing data for projected volume kube-api-access-667sr for pod openshift-network-diagnostics/network-check-target-h68z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:26.444297 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.443918    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr podName:a35a3ee0-24f0-473c-a046-2ef32cafb6a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:30.443901216 +0000 UTC m=+9.263193170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-667sr" (UniqueName: "kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr") pod "network-check-target-h68z9" (UID: "a35a3ee0-24f0-473c-a046-2ef32cafb6a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:26.713035 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:26.712115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:26.713035 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.712285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:26.713035 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:26.712683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:26.713035 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.712804    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:26.713035 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:26.712875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:26.713035 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:26.712944    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:28.712901 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:28.712469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:28.712901 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:28.712509    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:28.712901 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:28.712603    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:28.712901 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:28.712603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:28.712901 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:28.712709    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:28.712901 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:28.712809    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:30.385899 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:30.385859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:30.385899 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:30.385902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:30.386514 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.386027    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:30.386514 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.386094    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret podName:843bd172-4a4b-4285-8810-a1edee197936 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:38.386073826 +0000 UTC m=+17.205365785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret") pod "global-pull-secret-syncer-lsbm5" (UID: "843bd172-4a4b-4285-8810-a1edee197936") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:30.386514 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.386401    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:30.386514 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.386473    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:38.386452723 +0000 UTC m=+17.205744687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:30.486329 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:30.486221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:30.486484 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.486395    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:30.486484 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.486421    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:30.486484 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.486434    2577 projected.go:196] Error preparing data for projected volume kube-api-access-667sr for pod openshift-network-diagnostics/network-check-target-h68z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:30.486639 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.486499    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr podName:a35a3ee0-24f0-473c-a046-2ef32cafb6a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:38.48647959 +0000 UTC m=+17.305771544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-667sr" (UniqueName: "kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr") pod "network-check-target-h68z9" (UID: "a35a3ee0-24f0-473c-a046-2ef32cafb6a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:30.712423 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:30.711705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:30.712423 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.711867    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:30.712423 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:30.712228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:30.712423 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:30.712256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:30.712423 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.712334    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:30.712423 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:30.712390    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:32.712361 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:32.712335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:32.712701 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:32.712410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:32.712701 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:32.712509    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:32.712701 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:32.712512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:32.712701 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:32.712615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:32.712865 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:32.712705    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:34.712488 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:34.712461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:34.712856 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:34.712461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:34.712856 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:34.712583    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:34.712856 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:34.712679    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:34.712856 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:34.712461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:34.712856 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:34.712769    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:36.712153 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:36.712119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:36.712637 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:36.712122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:36.712637 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:36.712260    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:36.712637 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:36.712119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:36.712637 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:36.712322    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:36.712637 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:36.712398    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:38.448910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:38.448823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:38.448910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:38.448873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:38.449409 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.448997    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:38.449409 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.449003    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:38.449409 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.449058    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret podName:843bd172-4a4b-4285-8810-a1edee197936 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.449042912 +0000 UTC m=+33.268334852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret") pod "global-pull-secret-syncer-lsbm5" (UID: "843bd172-4a4b-4285-8810-a1edee197936") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:38.449409 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.449072    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.449065809 +0000 UTC m=+33.268357748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:38.549900 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:38.549859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:38.550088 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.550058    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:38.550088 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.550087    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:38.550174 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.550099    2577 projected.go:196] Error preparing data for projected volume kube-api-access-667sr for pod openshift-network-diagnostics/network-check-target-h68z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:38.550174 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.550159    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr podName:a35a3ee0-24f0-473c-a046-2ef32cafb6a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.550139451 +0000 UTC m=+33.369431434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-667sr" (UniqueName: "kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr") pod "network-check-target-h68z9" (UID: "a35a3ee0-24f0-473c-a046-2ef32cafb6a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:38.712928 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:38.712426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:38.712928 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:38.712537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:38.712928 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:38.712559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:38.712928 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.712554    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:38.712928 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.712634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:38.712928 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:38.712717    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:40.712251 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:40.712211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:40.712907 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:40.712295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:40.712907 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:40.712419    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:40.712907 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:40.712473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:40.712907 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:40.712561    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:40.712907 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:40.712655    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:42.712832 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.712075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:42.712832 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:42.712504    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:42.712832 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.712751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:42.712832 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.712139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:42.712832 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:42.712833    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:42.713417 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:42.712887    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:42.917831 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.917793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4g8x" event={"ID":"cb7e92bb-0e76-4a92-b79a-a74c6d4b751a","Type":"ContainerStarted","Data":"cc3ad8dd9b9fb7d70e8fd8f6636f5565ffab903528f2b7f29d3e54503e6de7f0"}
Sep 17 20:49:42.920287 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.920259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"d4e81b6fbd1dbc44c49bd90bf67b0ec1a19cbea7398717ce0a39b2ccba51dc4b"}
Sep 17 20:49:42.920395 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.920321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"97549cbc091942e2f755ea1bbebeb43516fd5a72be96526e767ae9a6e7adfbb7"}
Sep 17 20:49:42.920395 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.920358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"6f426181d7327fbece34dee3cac101d7007b832191368b135532581b72f7494e"}
Sep 17 20:49:42.920395 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.920389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"7889095a0dbee311c73440dd1dcf6c9859b1439a850a1c7714a6adcb05d417e9"}
Sep 17 20:49:42.921684 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.921659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fv4xg" event={"ID":"8ed1c2f5-88fd-453f-9b58-82b7e81fc60b","Type":"ContainerStarted","Data":"c632c61070f539a83234d4baef415a19ef66864986bf149eb0c1bb578d1d60c2"}
Sep 17 20:49:42.923158 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.923136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nzjvb" event={"ID":"72932197-8c1b-4d17-9e63-7cc9b1de8316","Type":"ContainerStarted","Data":"3db23bf20f4da58a9f253bd96cf86eea02527a5cd3e90fab2905b6e1ffa3a094"}
Sep 17 20:49:42.924541 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.924521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m6lqg" event={"ID":"49392876-888a-4002-a542-faa186562b69","Type":"ContainerStarted","Data":"dc12cd78ca360649bc7f7f8a91bfbc1c51876d12bcd135b2372f52b16cb368db"}
Sep 17 20:49:42.925872 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.925850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t" event={"ID":"72503c7a-e043-410b-b606-d4673b7c157c","Type":"ContainerStarted","Data":"77b9433b6e1cc0263098b1f3b19e2b1641e57a2267014ca9b8cb3e4c94581408"}
Sep 17 20:49:42.927201 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.927180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal" event={"ID":"43a2109c14934db412a36a19c6e1114f","Type":"ContainerStarted","Data":"eeab498bc8cc6276202f8cb47e104ef1bba6bfa80860a9e09230e811f343f131"}
Sep 17 20:49:42.927336 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.927322    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:42.928672 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.928652    2577 generic.go:356] "Generic (PLEG): container finished" podID="8722866d1367aa26c31e0552f2a8a7af" containerID="3f9a5ed00808af9200aaa273429a49be6e0da617e67dbdbe78e8bc9d1c1c09f5" exitCode=0
Sep 17 20:49:42.928777 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.928711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal" event={"ID":"8722866d1367aa26c31e0552f2a8a7af","Type":"ContainerDied","Data":"3f9a5ed00808af9200aaa273429a49be6e0da617e67dbdbe78e8bc9d1c1c09f5"}
Sep 17 20:49:42.930115 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.930095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h5ltl" event={"ID":"141c4051-4a21-4899-8871-eb7c00e693a2","Type":"ContainerStarted","Data":"15843abc1534789019ea00ae6bb176ccd929131b714b8dc819e30384e04b1b9c"}
Sep 17 20:49:42.931577 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.931548    2577 generic.go:356] "Generic (PLEG): container finished" podID="580f0aac-e36f-422d-b6dd-959e3b57d61f" containerID="c1fec53a397f41c89c0f9a2d1a6e8ed3e00e9f737831a73638265b860bb4854f" exitCode=0
Sep 17 20:49:42.931659 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.931581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerDied","Data":"c1fec53a397f41c89c0f9a2d1a6e8ed3e00e9f737831a73638265b860bb4854f"}
Sep 17 20:49:42.935999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.935960    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v4g8x" podStartSLOduration=2.558602877 podStartE2EDuration="21.935949302s" totalImagesPullingTime="19.377346425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:42.935570501 +0000 UTC m=+21.754862462" watchObservedRunningTime="2026-09-17 20:49:42.935949302 +0000 UTC m=+21.755241263"
Sep 17 20:49:42.938651 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.938636    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:42.938857 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.938842    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"]
Sep 17 20:49:42.959013 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.958972    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fv4xg" podStartSLOduration=2.648425625 podStartE2EDuration="21.958958579s" totalImagesPullingTime="19.310532954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:42.95867898 +0000 UTC m=+21.777970941" watchObservedRunningTime="2026-09-17 20:49:42.958958579 +0000 UTC m=+21.778250541"
Sep 17 20:49:42.990193 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:42.990141    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m6lqg" podStartSLOduration=6.917999455 podStartE2EDuration="21.990122041s" totalImagesPullingTime="15.072122586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:42.989712001 +0000 UTC m=+21.809003963" watchObservedRunningTime="2026-09-17 20:49:42.990122041 +0000 UTC m=+21.809414004"
Sep 17 20:49:43.045900 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.045856    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nzjvb" podStartSLOduration=2.814736624 podStartE2EDuration="22.045840534s" totalImagesPullingTime="19.23110391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.045634685 +0000 UTC m=+21.864926646" watchObservedRunningTime="2026-09-17 20:49:43.045840534 +0000 UTC m=+21.865132486"
Sep 17 20:49:43.063786 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.063737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h5ltl" podStartSLOduration=2.780240674 podStartE2EDuration="22.063722775s" totalImagesPullingTime="19.283482101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.063670492 +0000 UTC m=+21.882962453" watchObservedRunningTime="2026-09-17 20:49:43.063722775 +0000 UTC m=+21.883014737"
Sep 17 20:49:43.935403 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.935364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqsj2" event={"ID":"5afbfc91-924c-42d4-8e64-fcaac85986e4","Type":"ContainerStarted","Data":"29c45e2113679a88f09359101fb8d3b2a584497f1fa4c6019c18d890fea31fa3"}
Sep 17 20:49:43.942323 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.942211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"e03dd7641c310689779ffbd5006793cc1302b361ab24cd1dae1eb47596af5a6b"}
Sep 17 20:49:43.942323 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.942315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"7ac1625f996fba8cb867c690f976d0b180af170384d324e4a49405b437720fe8"}
Sep 17 20:49:43.945910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.945892    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:43.946205 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.946187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal" event={"ID":"8722866d1367aa26c31e0552f2a8a7af","Type":"ContainerStarted","Data":"673dfe59aa2639956aa1f61f2aad1b51f67959d60a0cbea7e5d92024afef0ed8"}
Sep 17 20:49:43.954635 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.954591    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal" podStartSLOduration=1.954574187 podStartE2EDuration="1.954574187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.953783657 +0000 UTC m=+22.773075619" watchObservedRunningTime="2026-09-17 20:49:43.954574187 +0000 UTC m=+22.773866150"
Sep 17 20:49:43.965469 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.965449    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:43.965561 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:43.965502    2577 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-35.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-35.ec2.internal"
Sep 17 20:49:43.997454 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.997409    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-35.ec2.internal" podStartSLOduration=21.997395063 podStartE2EDuration="21.997395063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.997295841 +0000 UTC m=+22.816587804" watchObservedRunningTime="2026-09-17 20:49:43.997395063 +0000 UTC m=+22.816687025"
Sep 17 20:49:43.997694 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:43.997658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gqsj2" podStartSLOduration=3.7529924709999998 podStartE2EDuration="22.997650216s" totalImagesPullingTime="19.244657745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.97846185 +0000 UTC m=+22.797753812" watchObservedRunningTime="2026-09-17 20:49:43.997650216 +0000 UTC m=+22.816942179"
Sep 17 20:49:44.058176 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.058146    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 20:49:44.692954 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.692628    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:49:44.058165Z","UUID":"92359047-1242-44ff-a6a4-376b22325ca3","Handler":null,"Name":"","Endpoint":""}
Sep 17 20:49:44.694773 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.694745    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 20:49:44.694909 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.694784    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 20:49:44.712269 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.712227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:44.712427 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:44.712355    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:44.712664 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.712645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:44.712758 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:44.712736    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:44.712811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.712788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:44.712882 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:44.712856    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:44.951156 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:44.951048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t" event={"ID":"72503c7a-e043-410b-b606-d4673b7c157c","Type":"ContainerStarted","Data":"b6adb0179c67905f1035f52c0ba4f49d03d52122f064f2af323d191153dcc8ae"}
Sep 17 20:49:45.957462 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:45.957415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"a8c18ac626da6419361e432b6bd71cbcaf8b261155fd215598e41ed59aac9e2a"}
Sep 17 20:49:45.959825 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:45.959797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t" event={"ID":"72503c7a-e043-410b-b606-d4673b7c157c","Type":"ContainerStarted","Data":"797ace2e6b668d6bd6d7fb286248a00f18365ce75e0e497c3b85eb6971070a00"}
Sep 17 20:49:45.978471 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:45.978426    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jf48t" podStartSLOduration=3.5604265760000002 podStartE2EDuration="24.978409533s" totalImagesPullingTime="21.417982957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:45.978383589 +0000 UTC m=+24.797675575" watchObservedRunningTime="2026-09-17 20:49:45.978409533 +0000 UTC m=+24.797701494"
Sep 17 20:49:46.179392 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:46.179354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:46.180059 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:46.180032    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:46.711906 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:46.711876    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:46.712069 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:46.711920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:46.712069 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:46.711981    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:46.712069 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:46.711920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:46.712069 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:46.712011    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:46.712221 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:46.712088    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:46.961758 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:46.961714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:46.962163 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:46.962026    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nzjvb"
Sep 17 20:49:47.965202 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:47.965170    2577 generic.go:356] "Generic (PLEG): container finished" podID="580f0aac-e36f-422d-b6dd-959e3b57d61f" containerID="3851e95ba6b3c7b0177f304ae0902e10344ec708271227194a5831a768c17b1c" exitCode=0
Sep 17 20:49:47.965694 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:47.965282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerDied","Data":"3851e95ba6b3c7b0177f304ae0902e10344ec708271227194a5831a768c17b1c"}
Sep 17 20:49:47.972706 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:47.972529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" event={"ID":"ff7b3b81-da9c-4d8c-8946-6d28cf2087e4","Type":"ContainerStarted","Data":"fc7094ee4502c1b9c9aaf493985e0a09c4ba6eaee0d8502e12498b21540ffc33"}
Sep 17 20:49:48.016443 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.016404    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f" podStartSLOduration=7.696831603 podStartE2EDuration="27.016390607s" totalImagesPullingTime="19.319559004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:48.013166562 +0000 UTC m=+26.832458523" watchObservedRunningTime="2026-09-17 20:49:48.016390607 +0000 UTC m=+26.835682568"
Sep 17 20:49:48.712473 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.712438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:48.712649 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.712442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:48.712649 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:48.712553    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:48.712649 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.712442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:48.712791 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:48.712673    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:48.712791 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:48.712700    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:48.974922 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.974846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:48.975303 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.974943    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:48.975303 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.974996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:48.990007 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.989979    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:48.990734 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:48.990700    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:49:49.476805 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.476775    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h68z9"]
Sep 17 20:49:49.476932 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.476874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:49.476984 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:49.476955    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:49.479569 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.479545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tl6x"]
Sep 17 20:49:49.479661 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.479648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:49.479765 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:49.479745    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:49.489381 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.489327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsbm5"]
Sep 17 20:49:49.489473 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.489415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:49.489533 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:49.489516    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:49.978137 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.978099    2577 generic.go:356] "Generic (PLEG): container finished" podID="580f0aac-e36f-422d-b6dd-959e3b57d61f" containerID="ade1b658fb72a44d93aaead04ce862338ee4923a1c85228794d8113fd72612fb" exitCode=0
Sep 17 20:49:49.978874 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:49.978188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerDied","Data":"ade1b658fb72a44d93aaead04ce862338ee4923a1c85228794d8113fd72612fb"}
Sep 17 20:49:50.712066 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:50.712038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:50.712212 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:50.712048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:50.712212 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:50.712127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:50.712314 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:50.712206    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:51.713118 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:51.712842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:51.713536 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:51.713177    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:51.984391 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:51.984325    2577 generic.go:356] "Generic (PLEG): container finished" podID="580f0aac-e36f-422d-b6dd-959e3b57d61f" containerID="66ce89162d1e039caf05d114aa15fe22402a9b38883a58a3d670c5e1ef541d54" exitCode=0
Sep 17 20:49:51.984391 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:51.984363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerDied","Data":"66ce89162d1e039caf05d114aa15fe22402a9b38883a58a3d670c5e1ef541d54"}
Sep 17 20:49:52.712317 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:52.712284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:52.712506 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:52.712386    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:52.712506 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:52.712439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:52.712630 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:52.712540    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:53.712277 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:53.712228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:53.712764 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:53.712391    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:54.465322 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:54.465229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:54.465322 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:54.465284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:54.465524 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.465346    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:54.465524 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.465376    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:54.465524 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.465420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.465400593 +0000 UTC m=+65.284692545 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:54.465524 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.465441    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret podName:843bd172-4a4b-4285-8810-a1edee197936 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.465430841 +0000 UTC m=+65.284722788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret") pod "global-pull-secret-syncer-lsbm5" (UID: "843bd172-4a4b-4285-8810-a1edee197936") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:54.565951 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:54.565919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:54.566136 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.566079    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:54.566136 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.566095    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:54.566136 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.566104    2577 projected.go:196] Error preparing data for projected volume kube-api-access-667sr for pod openshift-network-diagnostics/network-check-target-h68z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:54.566300 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.566161    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr podName:a35a3ee0-24f0-473c-a046-2ef32cafb6a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.566145004 +0000 UTC m=+65.385436960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-667sr" (UniqueName: "kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr") pod "network-check-target-h68z9" (UID: "a35a3ee0-24f0-473c-a046-2ef32cafb6a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:54.712396 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:54.712357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:54.712396 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:54.712386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:54.712929 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.712488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h68z9" podUID="a35a3ee0-24f0-473c-a046-2ef32cafb6a4"
Sep 17 20:49:54.712929 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:54.712633    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:49:55.711969 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:55.711883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:55.712184 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:55.712003    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsbm5" podUID="843bd172-4a4b-4285-8810-a1edee197936"
Sep 17 20:49:55.964649 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:55.964570    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-35.ec2.internal" event="NodeReady"
Sep 17 20:49:55.965071 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:55.964713    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 20:49:56.039397 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.039365    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mmt5q"]
Sep 17 20:49:56.055974 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.055953    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bb52p"]
Sep 17 20:49:56.056082 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.056068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.066118 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.066099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 20:49:56.066351 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.066335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 20:49:56.066463 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.066447    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b5xsj\""
Sep 17 20:49:56.069662 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.069649    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 20:49:56.073866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.073851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.081008 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.080984    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 20:49:56.081088 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.081030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 20:49:56.081088 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.081040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jjw4\""
Sep 17 20:49:56.089955 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.089931    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bb52p"]
Sep 17 20:49:56.094708 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.094688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mmt5q"]
Sep 17 20:49:56.179156 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.179127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qs8d\" (UniqueName: \"kubernetes.io/projected/a6630ca4-9c28-4f87-b38b-363fa2a0c764-kube-api-access-7qs8d\") pod \"ingress-canary-mmt5q\" (UID: \"a6630ca4-9c28-4f87-b38b-363fa2a0c764\") " pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.179308 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.179162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c18ebf4-2af7-4849-b66f-243015f1b7a4-config-volume\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.179308 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.179181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.179308 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.179196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c18ebf4-2af7-4849-b66f-243015f1b7a4-tmp-dir\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.179431 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.179309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6630ca4-9c28-4f87-b38b-363fa2a0c764-cert\") pod \"ingress-canary-mmt5q\" (UID: \"a6630ca4-9c28-4f87-b38b-363fa2a0c764\") " pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.179431 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.179330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmctr\" (UniqueName: \"kubernetes.io/projected/6c18ebf4-2af7-4849-b66f-243015f1b7a4-kube-api-access-vmctr\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.280480 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.280454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.280593 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.280482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c18ebf4-2af7-4849-b66f-243015f1b7a4-tmp-dir\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.280593 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.280540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6630ca4-9c28-4f87-b38b-363fa2a0c764-cert\") pod \"ingress-canary-mmt5q\" (UID: \"a6630ca4-9c28-4f87-b38b-363fa2a0c764\") " pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.280593 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.280565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmctr\" (UniqueName: \"kubernetes.io/projected/6c18ebf4-2af7-4849-b66f-243015f1b7a4-kube-api-access-vmctr\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.280699 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.280600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qs8d\" (UniqueName: \"kubernetes.io/projected/a6630ca4-9c28-4f87-b38b-363fa2a0c764-kube-api-access-7qs8d\") pod \"ingress-canary-mmt5q\" (UID: \"a6630ca4-9c28-4f87-b38b-363fa2a0c764\") " pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.280699 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:56.280607    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:56.280699 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:56.280680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.780658202 +0000 UTC m=+35.599950162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:56.280845 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.280703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c18ebf4-2af7-4849-b66f-243015f1b7a4-config-volume\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.280907 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.280886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c18ebf4-2af7-4849-b66f-243015f1b7a4-tmp-dir\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.281201 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.281182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c18ebf4-2af7-4849-b66f-243015f1b7a4-config-volume\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.285095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.285074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6630ca4-9c28-4f87-b38b-363fa2a0c764-cert\") pod \"ingress-canary-mmt5q\" (UID: \"a6630ca4-9c28-4f87-b38b-363fa2a0c764\") " pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.295152 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.295127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qs8d\" (UniqueName: \"kubernetes.io/projected/a6630ca4-9c28-4f87-b38b-363fa2a0c764-kube-api-access-7qs8d\") pod \"ingress-canary-mmt5q\" (UID: \"a6630ca4-9c28-4f87-b38b-363fa2a0c764\") " pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.296318 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.296226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmctr\" (UniqueName: \"kubernetes.io/projected/6c18ebf4-2af7-4849-b66f-243015f1b7a4-kube-api-access-vmctr\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.364339 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.364313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mmt5q"
Sep 17 20:49:56.703623 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.703432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mmt5q"]
Sep 17 20:49:56.707088 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:49:56.707059    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6630ca4_9c28_4f87_b38b_363fa2a0c764.slice/crio-83aedf0e7a6cf6c3da19cd6dd8ac36376ccffd3439a1053eec2f6fe5ce200a8b WatchSource:0}: Error finding container 83aedf0e7a6cf6c3da19cd6dd8ac36376ccffd3439a1053eec2f6fe5ce200a8b: Status 404 returned error can't find the container with id 83aedf0e7a6cf6c3da19cd6dd8ac36376ccffd3439a1053eec2f6fe5ce200a8b
Sep 17 20:49:56.711811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.711788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:49:56.711993 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.711973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:49:56.725273 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.725250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vnpsw\""
Sep 17 20:49:56.725674 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.725652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:49:56.726292 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.726277    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:49:56.727477 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.727460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j64n\""
Sep 17 20:49:56.727571 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.727507    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:49:56.783487 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.783458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:56.783634 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:56.783605    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:56.783674 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:56.783668    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:57.783650691 +0000 UTC m=+36.602942650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:56.999080 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.998995    2577 generic.go:356] "Generic (PLEG): container finished" podID="580f0aac-e36f-422d-b6dd-959e3b57d61f" containerID="9f490da1a5007898bac509535df15ed1f4b00515faf0a3b218224c806b69e841" exitCode=0
Sep 17 20:49:56.999568 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:56.999085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerDied","Data":"9f490da1a5007898bac509535df15ed1f4b00515faf0a3b218224c806b69e841"}
Sep 17 20:49:57.000230 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:57.000211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mmt5q" event={"ID":"a6630ca4-9c28-4f87-b38b-363fa2a0c764","Type":"ContainerStarted","Data":"83aedf0e7a6cf6c3da19cd6dd8ac36376ccffd3439a1053eec2f6fe5ce200a8b"}
Sep 17 20:49:57.715687 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:57.715649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:49:57.718527 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:57.718501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:49:57.791025 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:57.790994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:57.791191 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:57.791144    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:57.791376 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:57.791217    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:59.791199585 +0000 UTC m=+38.610491552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:58.006962 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:58.006924    2577 generic.go:356] "Generic (PLEG): container finished" podID="580f0aac-e36f-422d-b6dd-959e3b57d61f" containerID="91f9379310c726e0e65d0b377387b2eac08e79a1fac8cdb4353fbe43a5359c23" exitCode=0
Sep 17 20:49:58.007535 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:58.006990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerDied","Data":"91f9379310c726e0e65d0b377387b2eac08e79a1fac8cdb4353fbe43a5359c23"}
Sep 17 20:49:59.011016 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:59.010976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mmt5q" event={"ID":"a6630ca4-9c28-4f87-b38b-363fa2a0c764","Type":"ContainerStarted","Data":"e33537ddbc2ccdb7ea868c1397405a2421513966d47ffe81eaeda6afdf9d5b2c"}
Sep 17 20:49:59.014369 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:59.014337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" event={"ID":"580f0aac-e36f-422d-b6dd-959e3b57d61f","Type":"ContainerStarted","Data":"31f0b79483f87d60f478bc622e05adf1253757c0119dbb6cefd8ef20f3d3a937"}
Sep 17 20:49:59.079810 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:59.079762    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hfgtm" podStartSLOduration=7.999384754 podStartE2EDuration="38.079747163s" totalImagesPullingTime="30.080362409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:59.079554558 +0000 UTC m=+37.898846517" watchObservedRunningTime="2026-09-17 20:49:59.079747163 +0000 UTC m=+37.899039124"
Sep 17 20:49:59.079983 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:59.079875    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mmt5q" podStartSLOduration=1.049000018 podStartE2EDuration="3.079871792s" totalImagesPullingTime="2.030871774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:59.033274178 +0000 UTC m=+37.852566139" watchObservedRunningTime="2026-09-17 20:49:59.079871792 +0000 UTC m=+37.899163747"
Sep 17 20:49:59.806277 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:49:59.806064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:49:59.806446 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:59.806220    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:59.806446 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:49:59.806346    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:03.806329281 +0000 UTC m=+42.625621220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:03.833467 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:03.833421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:50:03.833885 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:03.833564    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:03.833885 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:03.833617    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:11.833603611 +0000 UTC m=+50.652895550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:11.893701 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:11.893662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:50:11.894182 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:11.893807    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:11.894182 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:11.893902    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:27.893885084 +0000 UTC m=+66.713177024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:13.948837 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.948800    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"]
Sep 17 20:50:13.951196 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.951180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:13.954726 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.954698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 20:50:13.954916 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.954897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 20:50:13.954991 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.954960    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 20:50:13.955724 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.955702    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 20:50:13.955826 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.955728    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 20:50:13.955885 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.955826    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 20:50:13.955934 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.955895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 20:50:13.964905 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:13.964877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"]
Sep 17 20:50:14.109544 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.109509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8pv\" (UniqueName: \"kubernetes.io/projected/401ab5f9-49f1-4d78-af57-c3576c5b352d-kube-api-access-rf8pv\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.109544 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.109548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.109764 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.109589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-hub\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.109764 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.109636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-ca\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.109764 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.109702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.109764 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.109735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ab5f9-49f1-4d78-af57-c3576c5b352d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.210077 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.210005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.210077 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.210052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-hub\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.210262 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.210076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-ca\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.210262 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.210129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.210262 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.210169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ab5f9-49f1-4d78-af57-c3576c5b352d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.210262 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.210206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8pv\" (UniqueName: \"kubernetes.io/projected/401ab5f9-49f1-4d78-af57-c3576c5b352d-kube-api-access-rf8pv\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.210904 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.210879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ab5f9-49f1-4d78-af57-c3576c5b352d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.213612 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.213589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-ca\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.213720 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.213695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.213787 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.213740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-hub\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.213787 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.213778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/401ab5f9-49f1-4d78-af57-c3576c5b352d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.218640 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.218609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8pv\" (UniqueName: \"kubernetes.io/projected/401ab5f9-49f1-4d78-af57-c3576c5b352d-kube-api-access-rf8pv\") pod \"cluster-proxy-proxy-agent-7dc84d8555-5zgf6\" (UID: \"401ab5f9-49f1-4d78-af57-c3576c5b352d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.281383 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.281343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:50:14.554356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:14.554325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"]
Sep 17 20:50:14.557450 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:50:14.557421    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod401ab5f9_49f1_4d78_af57_c3576c5b352d.slice/crio-8ec9cfeef133b582a0b18b1aa8151918f6f3b9c3f9f95f6ed40b0511cce1d2e6 WatchSource:0}: Error finding container 8ec9cfeef133b582a0b18b1aa8151918f6f3b9c3f9f95f6ed40b0511cce1d2e6: Status 404 returned error can't find the container with id 8ec9cfeef133b582a0b18b1aa8151918f6f3b9c3f9f95f6ed40b0511cce1d2e6
Sep 17 20:50:15.052091 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:15.052058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" event={"ID":"401ab5f9-49f1-4d78-af57-c3576c5b352d","Type":"ContainerStarted","Data":"8ec9cfeef133b582a0b18b1aa8151918f6f3b9c3f9f95f6ed40b0511cce1d2e6"}
Sep 17 20:50:18.062947 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:18.062908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" event={"ID":"401ab5f9-49f1-4d78-af57-c3576c5b352d","Type":"ContainerStarted","Data":"7517b13041ef5d64e3ca4092d485168030c1bf1bac35350a5a0af84aac647c34"}
Sep 17 20:50:18.063308 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:18.062957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" event={"ID":"401ab5f9-49f1-4d78-af57-c3576c5b352d","Type":"ContainerStarted","Data":"97d501b9043306fa7357709de1142494e0cf1c7913e797432343a5403edd8558"}
Sep 17 20:50:18.063308 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:18.062972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" event={"ID":"401ab5f9-49f1-4d78-af57-c3576c5b352d","Type":"ContainerStarted","Data":"9508c925e3c028ec0d98429d13637208f6b545c274751d205e226a86592798af"}
Sep 17 20:50:18.086305 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:18.086255    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" podStartSLOduration=1.788355905 podStartE2EDuration="5.086223084s" totalImagesPullingTime="3.297867179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:18.085550867 +0000 UTC m=+56.904842831" watchObservedRunningTime="2026-09-17 20:50:18.086223084 +0000 UTC m=+56.905515046"
Sep 17 20:50:20.990738 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:20.990710    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pdn6f"
Sep 17 20:50:26.505115 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.505079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:50:26.505115 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.505117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:50:26.508225 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.508206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:26.508461 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.508448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:26.515961 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:26.515945    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:50:26.516017 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:26.516008    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:51:30.515992572 +0000 UTC m=+129.335284513 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : secret "metrics-daemon-secret" not found
Sep 17 20:50:26.518182 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.518157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843bd172-4a4b-4285-8810-a1edee197936-original-pull-secret\") pod \"global-pull-secret-syncer-lsbm5\" (UID: \"843bd172-4a4b-4285-8810-a1edee197936\") " pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:50:26.526118 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.526098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsbm5"
Sep 17 20:50:26.605761 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.605728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:50:26.608589 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.608568    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:26.619016 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.618991    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:26.629337 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.629306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-667sr\" (UniqueName: \"kubernetes.io/projected/a35a3ee0-24f0-473c-a046-2ef32cafb6a4-kube-api-access-667sr\") pod \"network-check-target-h68z9\" (UID: \"a35a3ee0-24f0-473c-a046-2ef32cafb6a4\") " pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:50:26.725094 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.725061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vnpsw\""
Sep 17 20:50:26.733145 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.733125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:50:26.798106 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:26.798079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsbm5"]
Sep 17 20:50:26.801887 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:50:26.801862    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843bd172_4a4b_4285_8810_a1edee197936.slice/crio-b6def970af726e605233928893fc2ad2beef48c57c3c7b8904b99411b4de11d2 WatchSource:0}: Error finding container b6def970af726e605233928893fc2ad2beef48c57c3c7b8904b99411b4de11d2: Status 404 returned error can't find the container with id b6def970af726e605233928893fc2ad2beef48c57c3c7b8904b99411b4de11d2
Sep 17 20:50:27.004298 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:27.004268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h68z9"]
Sep 17 20:50:27.007424 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:50:27.007401    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35a3ee0_24f0_473c_a046_2ef32cafb6a4.slice/crio-73288ac88c610f0f158b221ccec4bff5141f3d664576dc7f990c777363ddce51 WatchSource:0}: Error finding container 73288ac88c610f0f158b221ccec4bff5141f3d664576dc7f990c777363ddce51: Status 404 returned error can't find the container with id 73288ac88c610f0f158b221ccec4bff5141f3d664576dc7f990c777363ddce51
Sep 17 20:50:27.085268 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:27.085164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsbm5" event={"ID":"843bd172-4a4b-4285-8810-a1edee197936","Type":"ContainerStarted","Data":"b6def970af726e605233928893fc2ad2beef48c57c3c7b8904b99411b4de11d2"}
Sep 17 20:50:27.086320 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:27.086292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h68z9" event={"ID":"a35a3ee0-24f0-473c-a046-2ef32cafb6a4","Type":"ContainerStarted","Data":"73288ac88c610f0f158b221ccec4bff5141f3d664576dc7f990c777363ddce51"}
Sep 17 20:50:27.912709 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:27.912672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:50:27.913201 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:27.912783    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:27.913201 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:27.912855    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:59.912837832 +0000 UTC m=+98.732129781 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:32.103298 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:32.103232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsbm5" event={"ID":"843bd172-4a4b-4285-8810-a1edee197936","Type":"ContainerStarted","Data":"4953b93647eb328e15fda9e19ca9bab973e6553d0103bfc5a9594c9a40586fcf"}
Sep 17 20:50:32.104693 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:32.104669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h68z9" event={"ID":"a35a3ee0-24f0-473c-a046-2ef32cafb6a4","Type":"ContainerStarted","Data":"184f8930b2dbcb240a8ca85878382adb87426714eb4fa45e5d969fb2b80a25b3"}
Sep 17 20:50:32.104814 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:32.104800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:50:32.123193 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:32.123151    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lsbm5" podStartSLOduration=66.426372753 podStartE2EDuration="1m11.123138217s" totalImagesPullingTime="4.696765464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:32.122894042 +0000 UTC m=+70.942186004" watchObservedRunningTime="2026-09-17 20:50:32.123138217 +0000 UTC m=+70.942430179"
Sep 17 20:50:59.936155 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:50:59.936118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:50:59.936622 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:59.936325    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:59.936622 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:50:59.936391    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls podName:6c18ebf4-2af7-4849-b66f-243015f1b7a4 nodeName:}" failed. No retries permitted until 2026-09-17 20:52:03.936374661 +0000 UTC m=+162.755666600 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls") pod "dns-default-bb52p" (UID: "6c18ebf4-2af7-4849-b66f-243015f1b7a4") : secret "dns-default-metrics-tls" not found
Sep 17 20:51:03.109744 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:03.109713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h68z9"
Sep 17 20:51:03.135428 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:03.135377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h68z9" podStartSLOduration=97.649453488 podStartE2EDuration="1m42.135359168s" totalImagesPullingTime="4.48590568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:32.141629494 +0000 UTC m=+70.960921456" watchObservedRunningTime="2026-09-17 20:51:03.135359168 +0000 UTC m=+101.954651124"
Sep 17 20:51:09.105201 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.105168    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hknz7"]
Sep 17 20:51:09.111077 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.111058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.113416 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.113396    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:51:09.113517 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.113462    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:51:09.113728 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.113711    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:51:09.113772 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.113742    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:51:09.114579 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.114561    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p9nvv\""
Sep 17 20:51:09.121098 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.121061    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hknz7"]
Sep 17 20:51:09.200709 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.200672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16adf-204a-4830-b44b-aac227bb8712-data-volume\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.200877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.200725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6f16adf-204a-4830-b44b-aac227bb8712-crio-socket\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.200877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.200751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.200877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.200770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqkz\" (UniqueName: \"kubernetes.io/projected/b6f16adf-204a-4830-b44b-aac227bb8712-kube-api-access-vmqkz\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.200877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.200818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.301877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.301840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6f16adf-204a-4830-b44b-aac227bb8712-crio-socket\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.301877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.301879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.302030 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.301899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqkz\" (UniqueName: \"kubernetes.io/projected/b6f16adf-204a-4830-b44b-aac227bb8712-kube-api-access-vmqkz\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.302030 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.301942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.302030 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.301979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6f16adf-204a-4830-b44b-aac227bb8712-crio-socket\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.302030 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.301987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16adf-204a-4830-b44b-aac227bb8712-data-volume\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.302030 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.302011    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.302204 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.302046    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.302204 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.302082    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.802064524 +0000 UTC m=+108.621356488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.302204 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.302107    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.80209439 +0000 UTC m=+108.621386363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.302382 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.302366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16adf-204a-4830-b44b-aac227bb8712-data-volume\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.313118 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.313099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqkz\" (UniqueName: \"kubernetes.io/projected/b6f16adf-204a-4830-b44b-aac227bb8712-kube-api-access-vmqkz\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.805904 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.805859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.806084 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.805978    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.806084 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.806053    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:10.806032257 +0000 UTC m=+109.625324220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.806213 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:09.806187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:09.806307 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.806273    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.806366 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:09.806336    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:10.806321139 +0000 UTC m=+109.625613092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.813336 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:10.813276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:10.813743 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:10.813412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:10.813743 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:10.813440    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.813743 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:10.813515    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.813495497 +0000 UTC m=+111.632787458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.813743 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:10.813522    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:10.813743 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:10.813572    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.813561276 +0000 UTC m=+111.632853216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:11.239774 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:11.239695    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h5ltl_141c4051-4a21-4899-8871-eb7c00e693a2/dns-node-resolver/0.log"
Sep 17 20:51:12.039199 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:12.039173    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m6lqg_49392876-888a-4002-a542-faa186562b69/node-ca/0.log"
Sep 17 20:51:12.641354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:12.641325    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mmt5q_a6630ca4-9c28-4f87-b38b-363fa2a0c764/serve-healthcheck-canary/0.log"
Sep 17 20:51:12.826919 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:12.826880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:12.827085 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:12.826930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:12.827085 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:12.827020    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:12.827085 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:12.827030    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:12.827085 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:12.827083    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.82706929 +0000 UTC m=+115.646361230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:12.827228 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:12.827096    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.827090363 +0000 UTC m=+115.646382302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:14.660751 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.660719    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85dd45785-s9mg2"]
Sep 17 20:51:14.663514 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.663499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.665901 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.665885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wszs2\""
Sep 17 20:51:14.666032 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.665904    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 20:51:14.666124 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.666106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 20:51:14.666203 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.666117    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 20:51:14.671857 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.671838    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 20:51:14.683072 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.683053    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85dd45785-s9mg2"]
Sep 17 20:51:14.738442 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-image-registry-private-configuration\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.738442 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-bound-sa-token\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.738658 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-certificates\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.738658 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-ca-trust-extracted\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.738658 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.738658 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-trusted-ca\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.738804 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-installation-pull-secrets\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.738804 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.738680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5tj\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-kube-api-access-fn5tj\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.838986 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.838951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-trusted-ca\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.838986 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.838989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-installation-pull-secrets\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839155 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5tj\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-kube-api-access-fn5tj\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839223 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-image-registry-private-configuration\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-bound-sa-token\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839347 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-certificates\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839400 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-ca-trust-extracted\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839400 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839521 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:14.839504    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:14.839576 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:14.839525    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd45785-s9mg2: secret "image-registry-tls" not found
Sep 17 20:51:14.839624 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:14.839591    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls podName:3737ae0b-0e29-4ffa-8d68-f87f4baa6a23 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:15.339572358 +0000 UTC m=+114.158864305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls") pod "image-registry-85dd45785-s9mg2" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23") : secret "image-registry-tls" not found
Sep 17 20:51:14.839769 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-ca-trust-extracted\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.839847 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-certificates\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.840013 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.839998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-trusted-ca\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.842102 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.842082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-installation-pull-secrets\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.842194 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.842086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-image-registry-private-configuration\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.851926 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.851900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-bound-sa-token\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:14.852010 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:14.851968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5tj\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-kube-api-access-fn5tj\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:15.343344 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:15.343293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:15.343530 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:15.343412    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:15.343530 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:15.343424    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd45785-s9mg2: secret "image-registry-tls" not found
Sep 17 20:51:15.343530 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:15.343487    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls podName:3737ae0b-0e29-4ffa-8d68-f87f4baa6a23 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.34347121 +0000 UTC m=+115.162763151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls") pod "image-registry-85dd45785-s9mg2" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23") : secret "image-registry-tls" not found
Sep 17 20:51:16.351142 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:16.351096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:16.351581 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:16.351276    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:16.351581 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:16.351294    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd45785-s9mg2: secret "image-registry-tls" not found
Sep 17 20:51:16.351581 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:16.351347    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls podName:3737ae0b-0e29-4ffa-8d68-f87f4baa6a23 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:18.351331478 +0000 UTC m=+117.170623419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls") pod "image-registry-85dd45785-s9mg2" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23") : secret "image-registry-tls" not found
Sep 17 20:51:16.855337 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:16.855304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:16.855495 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:16.855353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:16.855495 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:16.855432    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:16.855495 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:16.855448    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:16.855590 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:16.855506    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.855488977 +0000 UTC m=+123.674780923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:16.855590 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:16.855521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.855514507 +0000 UTC m=+123.674806447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:18.367046 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:18.367002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:18.367504 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:18.367187    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:18.367504 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:18.367207    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd45785-s9mg2: secret "image-registry-tls" not found
Sep 17 20:51:18.367504 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:18.367295    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls podName:3737ae0b-0e29-4ffa-8d68-f87f4baa6a23 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:22.367279072 +0000 UTC m=+121.186571011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls") pod "image-registry-85dd45785-s9mg2" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23") : secret "image-registry-tls" not found
Sep 17 20:51:22.397517 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:22.397477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:22.397896 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:22.397635    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:22.397896 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:22.397654    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd45785-s9mg2: secret "image-registry-tls" not found
Sep 17 20:51:22.397896 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:22.397709    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls podName:3737ae0b-0e29-4ffa-8d68-f87f4baa6a23 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:30.397694211 +0000 UTC m=+129.216986151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls") pod "image-registry-85dd45785-s9mg2" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23") : secret "image-registry-tls" not found
Sep 17 20:51:24.917810 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:24.917767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:24.918229 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:24.917822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:24.918229 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:24.917913    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.918229 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:24.917980    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls podName:b6f16adf-204a-4830-b44b-aac227bb8712 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:40.917962524 +0000 UTC m=+139.737254482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknz7" (UID: "b6f16adf-204a-4830-b44b-aac227bb8712") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.918376 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:24.918300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6f16adf-204a-4830-b44b-aac227bb8712-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:30.459447 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:30.459405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:30.461835 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:30.461815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"image-registry-85dd45785-s9mg2\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") " pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:30.560765 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:30.560731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:51:30.560947 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:30.560840    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:51:30.560947 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:30.560892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs podName:2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b nodeName:}" failed. No retries permitted until 2026-09-17 20:53:32.560877079 +0000 UTC m=+251.380169086 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs") pod "network-metrics-daemon-4tl6x" (UID: "2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b") : secret "metrics-daemon-secret" not found
Sep 17 20:51:30.575400 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:30.575377    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wszs2\""
Sep 17 20:51:30.583688 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:30.583663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:30.864918 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:30.864889    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85dd45785-s9mg2"]
Sep 17 20:51:30.867855 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:51:30.867832    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3737ae0b_0e29_4ffa_8d68_f87f4baa6a23.slice/crio-0adb02fdb5b4eb75dca882bf552a0b7779e6f7e5e8bb848ffa7d07e3bab4b78e WatchSource:0}: Error finding container 0adb02fdb5b4eb75dca882bf552a0b7779e6f7e5e8bb848ffa7d07e3bab4b78e: Status 404 returned error can't find the container with id 0adb02fdb5b4eb75dca882bf552a0b7779e6f7e5e8bb848ffa7d07e3bab4b78e
Sep 17 20:51:31.270569 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:31.270527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" event={"ID":"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23","Type":"ContainerStarted","Data":"052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76"}
Sep 17 20:51:31.270762 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:31.270591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" event={"ID":"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23","Type":"ContainerStarted","Data":"0adb02fdb5b4eb75dca882bf552a0b7779e6f7e5e8bb848ffa7d07e3bab4b78e"}
Sep 17 20:51:31.270762 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:31.270614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:51:31.292619 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:31.292563    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" podStartSLOduration=17.292543183 podStartE2EDuration="17.292543183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:31.291493906 +0000 UTC m=+130.110785867" watchObservedRunningTime="2026-09-17 20:51:31.292543183 +0000 UTC m=+130.111835146"
Sep 17 20:51:40.939759 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:40.939699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:40.942098 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:40.942080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6f16adf-204a-4830-b44b-aac227bb8712-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknz7\" (UID: \"b6f16adf-204a-4830-b44b-aac227bb8712\") " pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:41.221833 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:41.221753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p9nvv\""
Sep 17 20:51:41.230078 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:41.230061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hknz7"
Sep 17 20:51:41.507787 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:41.507759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hknz7"]
Sep 17 20:51:41.510672 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:51:41.510645    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f16adf_204a_4830_b44b_aac227bb8712.slice/crio-046ac7164c818243353206abb0829749427e21f80a65860b377329b27c021c59 WatchSource:0}: Error finding container 046ac7164c818243353206abb0829749427e21f80a65860b377329b27c021c59: Status 404 returned error can't find the container with id 046ac7164c818243353206abb0829749427e21f80a65860b377329b27c021c59
Sep 17 20:51:42.304665 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:42.304638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknz7" event={"ID":"b6f16adf-204a-4830-b44b-aac227bb8712","Type":"ContainerStarted","Data":"56106900d14ce3c0689e093eced1ad296f79273484c4b407ac3530fdb6569930"}
Sep 17 20:51:42.304947 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:42.304697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknz7" event={"ID":"b6f16adf-204a-4830-b44b-aac227bb8712","Type":"ContainerStarted","Data":"046ac7164c818243353206abb0829749427e21f80a65860b377329b27c021c59"}
Sep 17 20:51:43.309807 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:43.309766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknz7" event={"ID":"b6f16adf-204a-4830-b44b-aac227bb8712","Type":"ContainerStarted","Data":"6e91c11f9b61fd070d6b5464519f023a95b829a2b2285b925ac969e84cafb9d2"}
Sep 17 20:51:44.283401 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:44.283359    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" podUID="401ab5f9-49f1-4d78-af57-c3576c5b352d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 20:51:44.316391 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:44.316359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknz7" event={"ID":"b6f16adf-204a-4830-b44b-aac227bb8712","Type":"ContainerStarted","Data":"c5fe653c11f96c5af7d52b0eac4182db91c3fa572ea95ba78237f8ee553b9598"}
Sep 17 20:51:44.336641 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:44.336591    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hknz7" podStartSLOduration=33.397617212 podStartE2EDuration="35.336577153s" totalImagesPullingTime="1.938959941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:44.33581522 +0000 UTC m=+143.155107182" watchObservedRunningTime="2026-09-17 20:51:44.336577153 +0000 UTC m=+143.155869115"
Sep 17 20:51:47.114251 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.114215    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85dd45785-s9mg2"]
Sep 17 20:51:47.118343 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.118305    2577 patch_prober.go:28] interesting pod/image-registry-85dd45785-s9mg2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 20:51:47.118479 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.118354    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" podUID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 20:51:47.211119 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.211079    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-554db46f86-nqcg6"]
Sep 17 20:51:47.213992 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.213977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.260361 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.260330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-554db46f86-nqcg6"]
Sep 17 20:51:47.285601 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxv4\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-kube-api-access-9mxv4\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.285601 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b241ce8-0d29-4e6f-8713-fff56fcf9774-ca-trust-extracted\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.285765 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-registry-tls\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.285765 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b241ce8-0d29-4e6f-8713-fff56fcf9774-installation-pull-secrets\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.285765 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b241ce8-0d29-4e6f-8713-fff56fcf9774-trusted-ca\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.285765 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-bound-sa-token\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.285765 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b241ce8-0d29-4e6f-8713-fff56fcf9774-registry-certificates\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.285915 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.285782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b241ce8-0d29-4e6f-8713-fff56fcf9774-image-registry-private-configuration\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386482 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b241ce8-0d29-4e6f-8713-fff56fcf9774-registry-certificates\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386482 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b241ce8-0d29-4e6f-8713-fff56fcf9774-image-registry-private-configuration\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386482 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxv4\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-kube-api-access-9mxv4\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386482 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b241ce8-0d29-4e6f-8713-fff56fcf9774-ca-trust-extracted\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386788 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-registry-tls\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386788 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b241ce8-0d29-4e6f-8713-fff56fcf9774-installation-pull-secrets\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386788 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b241ce8-0d29-4e6f-8713-fff56fcf9774-trusted-ca\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386788 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-bound-sa-token\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.386994 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.386870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b241ce8-0d29-4e6f-8713-fff56fcf9774-ca-trust-extracted\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.387371 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.387332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b241ce8-0d29-4e6f-8713-fff56fcf9774-registry-certificates\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.387598 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.387581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b241ce8-0d29-4e6f-8713-fff56fcf9774-trusted-ca\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.389205 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.389186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b241ce8-0d29-4e6f-8713-fff56fcf9774-installation-pull-secrets\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.389524 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.389510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b241ce8-0d29-4e6f-8713-fff56fcf9774-image-registry-private-configuration\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.389647 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.389628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-registry-tls\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.402452 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.402423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-bound-sa-token\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.402905 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.402879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxv4\" (UniqueName: \"kubernetes.io/projected/4b241ce8-0d29-4e6f-8713-fff56fcf9774-kube-api-access-9mxv4\") pod \"image-registry-554db46f86-nqcg6\" (UID: \"4b241ce8-0d29-4e6f-8713-fff56fcf9774\") " pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.522303 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.522277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:47.796583 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:47.796549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-554db46f86-nqcg6"]
Sep 17 20:51:47.800300 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:51:47.800262    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b241ce8_0d29_4e6f_8713_fff56fcf9774.slice/crio-2cb156e2437f72e00d0288689bfcf7528f0708b8e429806dd3f4846fe5b44410 WatchSource:0}: Error finding container 2cb156e2437f72e00d0288689bfcf7528f0708b8e429806dd3f4846fe5b44410: Status 404 returned error can't find the container with id 2cb156e2437f72e00d0288689bfcf7528f0708b8e429806dd3f4846fe5b44410
Sep 17 20:51:48.329206 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:48.329170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554db46f86-nqcg6" event={"ID":"4b241ce8-0d29-4e6f-8713-fff56fcf9774","Type":"ContainerStarted","Data":"e2535cdc167153b3073052f5670f7fe661140d1b669e72f52199b071ea00e5c6"}
Sep 17 20:51:48.329589 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:48.329217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554db46f86-nqcg6" event={"ID":"4b241ce8-0d29-4e6f-8713-fff56fcf9774","Type":"ContainerStarted","Data":"2cb156e2437f72e00d0288689bfcf7528f0708b8e429806dd3f4846fe5b44410"}
Sep 17 20:51:48.329589 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:48.329266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:51:48.352031 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:48.351977    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-554db46f86-nqcg6" podStartSLOduration=1.351962071 podStartE2EDuration="1.351962071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:48.350837618 +0000 UTC m=+147.170129591" watchObservedRunningTime="2026-09-17 20:51:48.351962071 +0000 UTC m=+147.171254032"
Sep 17 20:51:53.634111 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.634080    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9jzjk"]
Sep 17 20:51:53.639264 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.639227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.646404 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.646386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 20:51:53.646996 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.646980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 20:51:53.647272 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.647257    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 20:51:53.647332 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.647296    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 20:51:53.647377 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.647294    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 20:51:53.648578 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.648564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8f996\""
Sep 17 20:51:53.649522 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.649510    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 20:51:53.733013 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.732983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxk9\" (UniqueName: \"kubernetes.io/projected/620dcb8e-480f-4815-8152-695cdf9e9ef0-kube-api-access-7pxk9\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733186 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-textfile\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733186 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-sys\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733186 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-root\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733186 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733186 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733436 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-wtmp\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733436 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/620dcb8e-480f-4815-8152-695cdf9e9ef0-metrics-client-ca\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.733436 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.733285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-tls\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834657 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834657 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-wtmp\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/620dcb8e-480f-4815-8152-695cdf9e9ef0-metrics-client-ca\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-tls\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxk9\" (UniqueName: \"kubernetes.io/projected/620dcb8e-480f-4815-8152-695cdf9e9ef0-kube-api-access-7pxk9\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-textfile\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-sys\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:53.834820    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-root\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.834924 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:53.834890    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-tls podName:620dcb8e-480f-4815-8152-695cdf9e9ef0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:54.33486766 +0000 UTC m=+153.154159602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-tls") pod "node-exporter-9jzjk" (UID: "620dcb8e-480f-4815-8152-695cdf9e9ef0") : secret "node-exporter-tls" not found
Sep 17 20:51:53.835428 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-wtmp\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.835428 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.834954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-sys\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.835428 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.835012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/620dcb8e-480f-4815-8152-695cdf9e9ef0-root\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.835428 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.835161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-textfile\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.835428 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.835316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.835428 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.835417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/620dcb8e-480f-4815-8152-695cdf9e9ef0-metrics-client-ca\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.837143 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.837123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:53.848026 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:53.847998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxk9\" (UniqueName: \"kubernetes.io/projected/620dcb8e-480f-4815-8152-695cdf9e9ef0-kube-api-access-7pxk9\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:54.282870 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:54.282829    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" podUID="401ab5f9-49f1-4d78-af57-c3576c5b352d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 20:51:54.339146 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:54.339114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-tls\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:54.341475 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:54.341459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/620dcb8e-480f-4815-8152-695cdf9e9ef0-node-exporter-tls\") pod \"node-exporter-9jzjk\" (UID: \"620dcb8e-480f-4815-8152-695cdf9e9ef0\") " pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:54.548649 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:54.548566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9jzjk"
Sep 17 20:51:54.559166 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:51:54.559144    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620dcb8e_480f_4815_8152_695cdf9e9ef0.slice/crio-4cef685d0e234471bc01422ba5ddd2bb2b8e7e3814d5fe146474aa706fc348d9 WatchSource:0}: Error finding container 4cef685d0e234471bc01422ba5ddd2bb2b8e7e3814d5fe146474aa706fc348d9: Status 404 returned error can't find the container with id 4cef685d0e234471bc01422ba5ddd2bb2b8e7e3814d5fe146474aa706fc348d9
Sep 17 20:51:55.351607 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:55.351576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jzjk" event={"ID":"620dcb8e-480f-4815-8152-695cdf9e9ef0","Type":"ContainerStarted","Data":"4cef685d0e234471bc01422ba5ddd2bb2b8e7e3814d5fe146474aa706fc348d9"}
Sep 17 20:51:56.355777 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:56.355743    2577 generic.go:356] "Generic (PLEG): container finished" podID="620dcb8e-480f-4815-8152-695cdf9e9ef0" containerID="d5f796f7ce14a16c1d303cb218c74998cde2901d772e01f88708f35587a09181" exitCode=0
Sep 17 20:51:56.356157 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:56.355794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jzjk" event={"ID":"620dcb8e-480f-4815-8152-695cdf9e9ef0","Type":"ContainerDied","Data":"d5f796f7ce14a16c1d303cb218c74998cde2901d772e01f88708f35587a09181"}
Sep 17 20:51:57.119141 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:57.119105    2577 patch_prober.go:28] interesting pod/image-registry-85dd45785-s9mg2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 20:51:57.119327 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:57.119157    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" podUID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 20:51:57.360913 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:57.360879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jzjk" event={"ID":"620dcb8e-480f-4815-8152-695cdf9e9ef0","Type":"ContainerStarted","Data":"1a03a3e0887c8407f5731a1edd9c645a4169d24da83c57193211b6272d48394b"}
Sep 17 20:51:57.361295 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:57.360957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jzjk" event={"ID":"620dcb8e-480f-4815-8152-695cdf9e9ef0","Type":"ContainerStarted","Data":"57e2f65de291e7d752d907e90506bca5e8ba29202a24778ed6801024c8c4adf6"}
Sep 17 20:51:57.382978 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:57.382880    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9jzjk" podStartSLOduration=3.5015746979999998 podStartE2EDuration="4.382865293s" totalImagesPullingTime="812.608795ms" totalInitContainerRuntime="68.6818ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:57.381838453 +0000 UTC m=+156.201130414" watchObservedRunningTime="2026-09-17 20:51:57.382865293 +0000 UTC m=+156.202157251"
Sep 17 20:51:59.081172 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:59.081130    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bb52p" podUID="6c18ebf4-2af7-4849-b66f-243015f1b7a4"
Sep 17 20:51:59.367152 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:51:59.367076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bb52p"
Sep 17 20:51:59.727937 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:51:59.727854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4tl6x" podUID="2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b"
Sep 17 20:52:04.015416 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.015376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:52:04.017717 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.017699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c18ebf4-2af7-4849-b66f-243015f1b7a4-metrics-tls\") pod \"dns-default-bb52p\" (UID: \"6c18ebf4-2af7-4849-b66f-243015f1b7a4\") " pod="openshift-dns/dns-default-bb52p"
Sep 17 20:52:04.170915 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.170880    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jjw4\""
Sep 17 20:52:04.179267 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.179246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bb52p"
Sep 17 20:52:04.282437 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.282360    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" podUID="401ab5f9-49f1-4d78-af57-c3576c5b352d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 20:52:04.282553 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.282448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6"
Sep 17 20:52:04.282920 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.282877    2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"7517b13041ef5d64e3ca4092d485168030c1bf1bac35350a5a0af84aac647c34"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 20:52:04.282955 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.282941    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" podUID="401ab5f9-49f1-4d78-af57-c3576c5b352d" containerName="service-proxy" containerID="cri-o://7517b13041ef5d64e3ca4092d485168030c1bf1bac35350a5a0af84aac647c34" gracePeriod=30
Sep 17 20:52:04.452120 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:04.452094    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bb52p"]
Sep 17 20:52:04.456172 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:52:04.456146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c18ebf4_2af7_4849_b66f_243015f1b7a4.slice/crio-38af76f9dfa70e4d90067580b300d28fd04e617c6aaa54b609413ae03078f0dd WatchSource:0}: Error finding container 38af76f9dfa70e4d90067580b300d28fd04e617c6aaa54b609413ae03078f0dd: Status 404 returned error can't find the container with id 38af76f9dfa70e4d90067580b300d28fd04e617c6aaa54b609413ae03078f0dd
Sep 17 20:52:05.387818 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:05.387787    2577 generic.go:356] "Generic (PLEG): container finished" podID="401ab5f9-49f1-4d78-af57-c3576c5b352d" containerID="7517b13041ef5d64e3ca4092d485168030c1bf1bac35350a5a0af84aac647c34" exitCode=2
Sep 17 20:52:05.388225 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:05.387862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" event={"ID":"401ab5f9-49f1-4d78-af57-c3576c5b352d","Type":"ContainerDied","Data":"7517b13041ef5d64e3ca4092d485168030c1bf1bac35350a5a0af84aac647c34"}
Sep 17 20:52:05.388225 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:05.387903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dc84d8555-5zgf6" event={"ID":"401ab5f9-49f1-4d78-af57-c3576c5b352d","Type":"ContainerStarted","Data":"bce06720471c1d4404aee998ff623cda82a6db631dfb3c1240d5c52259f8ca0f"}
Sep 17 20:52:05.389011 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:05.388993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bb52p" event={"ID":"6c18ebf4-2af7-4849-b66f-243015f1b7a4","Type":"ContainerStarted","Data":"38af76f9dfa70e4d90067580b300d28fd04e617c6aaa54b609413ae03078f0dd"}
Sep 17 20:52:06.394047 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:06.394010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bb52p" event={"ID":"6c18ebf4-2af7-4849-b66f-243015f1b7a4","Type":"ContainerStarted","Data":"617bcc5a0862faa251cbbf2eafbde6fa4e912d8d794ba7436aeedf594ac83393"}
Sep 17 20:52:06.394442 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:06.394084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bb52p" event={"ID":"6c18ebf4-2af7-4849-b66f-243015f1b7a4","Type":"ContainerStarted","Data":"b57fb51339b5dfeef7ed5c71a044bb91ded10b2a78cf49727243b1d5e2ad6b10"}
Sep 17 20:52:06.394442 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:06.394153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bb52p"
Sep 17 20:52:06.415452 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:06.415402    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bb52p" podStartSLOduration=128.991305194 podStartE2EDuration="2m10.415388477s" totalImagesPullingTime="1.424083283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:06.414089127 +0000 UTC m=+165.233381088" watchObservedRunningTime="2026-09-17 20:52:06.415388477 +0000 UTC m=+165.234680438"
Sep 17 20:52:07.118696 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.118669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:52:07.434427 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.434355    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b95d48c86-xlflc"]
Sep 17 20:52:07.437390 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.437375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.440578 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.440558    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 20:52:07.440578 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.440574    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-srgxm\""
Sep 17 20:52:07.440734 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.440557    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 20:52:07.440827 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.440813    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 20:52:07.441552 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.441534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 20:52:07.441700 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.441688    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 20:52:07.441899 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.441888    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 20:52:07.442041 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.442030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 20:52:07.490479 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.490449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b95d48c86-xlflc"]
Sep 17 20:52:07.526567 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.526534    2577 patch_prober.go:28] interesting pod/image-registry-554db46f86-nqcg6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 20:52:07.526724 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.526592    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-554db46f86-nqcg6" podUID="4b241ce8-0d29-4e6f-8713-fff56fcf9774" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 20:52:07.544790 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.544761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-oauth-config\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.544921 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.544802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbxv\" (UniqueName: \"kubernetes.io/projected/9bd1252a-1b63-45c5-8689-85fa8de3b532-kube-api-access-kjbxv\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.544921 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.544861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-config\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.544921 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.544917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-serving-cert\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.545047 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.544940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-oauth-serving-cert\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.545047 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.544957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-service-ca\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.645420 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.645376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-config\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.645596 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.645436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-serving-cert\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.645596 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.645457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-oauth-serving-cert\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.645596 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.645476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-service-ca\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.645596 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.645585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-oauth-config\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.645789 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.645637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbxv\" (UniqueName: \"kubernetes.io/projected/9bd1252a-1b63-45c5-8689-85fa8de3b532-kube-api-access-kjbxv\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.646194 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.646129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-config\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.646194 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.646200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-service-ca\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.646498 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.646380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-oauth-serving-cert\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.647999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.647972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-oauth-config\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.648153 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.648135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-serving-cert\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.657200 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.657178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbxv\" (UniqueName: \"kubernetes.io/projected/9bd1252a-1b63-45c5-8689-85fa8de3b532-kube-api-access-kjbxv\") pod \"console-7b95d48c86-xlflc\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") " pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:07.746411 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:07.746322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:08.042715 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:08.042687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b95d48c86-xlflc"]
Sep 17 20:52:08.045943 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:52:08.045919    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd1252a_1b63_45c5_8689_85fa8de3b532.slice/crio-4efbb0be32b475bc881488c1289cce0fe23de8df681ddc2b9fcc23728b951a72 WatchSource:0}: Error finding container 4efbb0be32b475bc881488c1289cce0fe23de8df681ddc2b9fcc23728b951a72: Status 404 returned error can't find the container with id 4efbb0be32b475bc881488c1289cce0fe23de8df681ddc2b9fcc23728b951a72
Sep 17 20:52:08.401799 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:08.401715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95d48c86-xlflc" event={"ID":"9bd1252a-1b63-45c5-8689-85fa8de3b532","Type":"ContainerStarted","Data":"4efbb0be32b475bc881488c1289cce0fe23de8df681ddc2b9fcc23728b951a72"}
Sep 17 20:52:09.338673 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:09.338635    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-554db46f86-nqcg6"
Sep 17 20:52:12.133439 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.133362    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" podUID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" containerName="registry" containerID="cri-o://052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76" gracePeriod=30
Sep 17 20:52:12.379645 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.379623    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:52:12.417780 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.417751    2577 generic.go:356] "Generic (PLEG): container finished" podID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" containerID="052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76" exitCode=0
Sep 17 20:52:12.417939 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.417808    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd45785-s9mg2"
Sep 17 20:52:12.417939 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.417840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" event={"ID":"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23","Type":"ContainerDied","Data":"052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76"}
Sep 17 20:52:12.417939 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.417893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd45785-s9mg2" event={"ID":"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23","Type":"ContainerDied","Data":"0adb02fdb5b4eb75dca882bf552a0b7779e6f7e5e8bb848ffa7d07e3bab4b78e"}
Sep 17 20:52:12.417939 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.417932    2577 scope.go:117] "RemoveContainer" containerID="052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76"
Sep 17 20:52:12.419525 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.419498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95d48c86-xlflc" event={"ID":"9bd1252a-1b63-45c5-8689-85fa8de3b532","Type":"ContainerStarted","Data":"0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a"}
Sep 17 20:52:12.425810 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.425794    2577 scope.go:117] "RemoveContainer" containerID="052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76"
Sep 17 20:52:12.426091 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:52:12.426073    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76\": container with ID starting with 052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76 not found: ID does not exist" containerID="052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76"
Sep 17 20:52:12.426152 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.426103    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76"} err="failed to get container status \"052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76\": rpc error: code = NotFound desc = could not find container \"052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76\": container with ID starting with 052f127b2b753fcef751b4a8394eaa3a441ceabfcbb761559d8212c33e4e1c76 not found: ID does not exist"
Sep 17 20:52:12.439889 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.439847    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b95d48c86-xlflc" podStartSLOduration=1.562468386 podStartE2EDuration="5.439830694s" totalImagesPullingTime="3.877362308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:12.439351613 +0000 UTC m=+171.258643575" watchObservedRunningTime="2026-09-17 20:52:12.439830694 +0000 UTC m=+171.259122657"
Sep 17 20:52:12.493086 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493060    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.493268 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-trusted-ca\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.493268 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-bound-sa-token\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.493268 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-ca-trust-extracted\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.493268 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn5tj\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-kube-api-access-fn5tj\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.493268 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-certificates\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.493505 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-installation-pull-secrets\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.493505 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-image-registry-private-configuration\") pod \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\" (UID: \"3737ae0b-0e29-4ffa-8d68-f87f4baa6a23\") "
Sep 17 20:52:12.494117 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.493676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:12.494117 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.494034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:12.495989 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.495947    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-kube-api-access-fn5tj" (OuterVolumeSpecName: "kube-api-access-fn5tj") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "kube-api-access-fn5tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:12.496146 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.496119    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:12.496225 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.496141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:12.496393 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.496339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:12.496468 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.496444    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:12.501992 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.501943    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" (UID: "3737ae0b-0e29-4ffa-8d68-f87f4baa6a23"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:52:12.594324 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594281    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-bound-sa-token\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.594324 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594319    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-ca-trust-extracted\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.594324 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594332    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5tj\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-kube-api-access-fn5tj\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.594324 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594340    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-certificates\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.594555 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594349    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-installation-pull-secrets\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.594555 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594360    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-image-registry-private-configuration\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.594555 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594368    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-registry-tls\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.594555 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.594376    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23-trusted-ca\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:12.741353 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.741317    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85dd45785-s9mg2"]
Sep 17 20:52:12.744886 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:12.744861    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85dd45785-s9mg2"]
Sep 17 20:52:13.712430 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:13.712348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:52:13.716442 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:13.716420    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" path="/var/lib/kubelet/pods/3737ae0b-0e29-4ffa-8d68-f87f4baa6a23/volumes"
Sep 17 20:52:16.127303 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:16.127230    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bb52p_6c18ebf4-2af7-4849-b66f-243015f1b7a4/dns/0.log"
Sep 17 20:52:16.325255 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:16.325217    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bb52p_6c18ebf4-2af7-4849-b66f-243015f1b7a4/kube-rbac-proxy/0.log"
Sep 17 20:52:16.399431 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:16.399366    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bb52p"
Sep 17 20:52:17.324777 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.324743    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h5ltl_141c4051-4a21-4899-8871-eb7c00e693a2/dns-node-resolver/0.log"
Sep 17 20:52:17.416018 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.415987    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f588dbff9-9ztdp"]
Sep 17 20:52:17.416261 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.416231    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" containerName="registry"
Sep 17 20:52:17.416311 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.416263    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" containerName="registry"
Sep 17 20:52:17.416345 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.416311    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3737ae0b-0e29-4ffa-8d68-f87f4baa6a23" containerName="registry"
Sep 17 20:52:17.420747 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.420725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.429001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.428982    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 20:52:17.432578 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.432557    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f588dbff9-9ztdp"]
Sep 17 20:52:17.534659 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.534628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-console-config\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.534814 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.534673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-service-ca\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.534814 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.534716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-oauth-config\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.534814 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.534745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-oauth-serving-cert\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.534814 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.534784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-trusted-ca-bundle\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.534970 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.534823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-serving-cert\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.534970 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.534879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cll7d\" (UniqueName: \"kubernetes.io/projected/2ce5dffe-fb11-429c-b945-f83041733d69-kube-api-access-cll7d\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.635260 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.635152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-trusted-ca-bundle\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.635260 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.635195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-serving-cert\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.635260 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.635218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cll7d\" (UniqueName: \"kubernetes.io/projected/2ce5dffe-fb11-429c-b945-f83041733d69-kube-api-access-cll7d\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.635260 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.635260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-console-config\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.635551 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.635286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-service-ca\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.635551 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.635349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-oauth-config\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.635551 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.635386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-oauth-serving-cert\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.636095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.636070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-service-ca\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.636095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.636089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-console-config\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.636229 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.636098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-oauth-serving-cert\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.636409 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.636385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-trusted-ca-bundle\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.637798 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.637772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-serving-cert\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.637876 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.637802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-oauth-config\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.648852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.648834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cll7d\" (UniqueName: \"kubernetes.io/projected/2ce5dffe-fb11-429c-b945-f83041733d69-kube-api-access-cll7d\") pod \"console-5f588dbff9-9ztdp\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") " pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.731150 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.731115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:17.746931 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.746891    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:17.747035 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.746966    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:17.752152 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.752131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:17.931561 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:17.931456    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mmt5q_a6630ca4-9c28-4f87-b38b-363fa2a0c764/serve-healthcheck-canary/0.log"
Sep 17 20:52:18.033661 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:52:18.033630    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce5dffe_fb11_429c_b945_f83041733d69.slice/crio-24a2c13e18dcf396ddcff9427f454feae159a9e03685da48db733365c5a13377 WatchSource:0}: Error finding container 24a2c13e18dcf396ddcff9427f454feae159a9e03685da48db733365c5a13377: Status 404 returned error can't find the container with id 24a2c13e18dcf396ddcff9427f454feae159a9e03685da48db733365c5a13377
Sep 17 20:52:18.038506 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:18.038485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f588dbff9-9ztdp"]
Sep 17 20:52:18.443018 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:18.442984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f588dbff9-9ztdp" event={"ID":"2ce5dffe-fb11-429c-b945-f83041733d69","Type":"ContainerStarted","Data":"e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e"}
Sep 17 20:52:18.443543 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:18.443032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f588dbff9-9ztdp" event={"ID":"2ce5dffe-fb11-429c-b945-f83041733d69","Type":"ContainerStarted","Data":"24a2c13e18dcf396ddcff9427f454feae159a9e03685da48db733365c5a13377"}
Sep 17 20:52:18.446824 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:18.446793    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:18.463825 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:18.463771    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f588dbff9-9ztdp" podStartSLOduration=1.463755617 podStartE2EDuration="1.463755617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:18.462223733 +0000 UTC m=+177.281515694" watchObservedRunningTime="2026-09-17 20:52:18.463755617 +0000 UTC m=+177.283047578"
Sep 17 20:52:27.731584 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:27.731550    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:27.731958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:27.731612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:27.736307 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:27.736285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:28.479949 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:28.479920    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:52:28.532170 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:28.532138    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b95d48c86-xlflc"]
Sep 17 20:52:53.551447 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.551378    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b95d48c86-xlflc" podUID="9bd1252a-1b63-45c5-8689-85fa8de3b532" containerName="console" containerID="cri-o://0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a" gracePeriod=15
Sep 17 20:52:53.788303 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.788280    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:53.914390 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914314    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-config\") pod \"9bd1252a-1b63-45c5-8689-85fa8de3b532\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") "
Sep 17 20:52:53.914524 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-oauth-serving-cert\") pod \"9bd1252a-1b63-45c5-8689-85fa8de3b532\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") "
Sep 17 20:52:53.914524 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914454    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-service-ca\") pod \"9bd1252a-1b63-45c5-8689-85fa8de3b532\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") "
Sep 17 20:52:53.914524 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjbxv\" (UniqueName: \"kubernetes.io/projected/9bd1252a-1b63-45c5-8689-85fa8de3b532-kube-api-access-kjbxv\") pod \"9bd1252a-1b63-45c5-8689-85fa8de3b532\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") "
Sep 17 20:52:53.914524 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914515    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-serving-cert\") pod \"9bd1252a-1b63-45c5-8689-85fa8de3b532\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") "
Sep 17 20:52:53.914721 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914533    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-oauth-config\") pod \"9bd1252a-1b63-45c5-8689-85fa8de3b532\" (UID: \"9bd1252a-1b63-45c5-8689-85fa8de3b532\") "
Sep 17 20:52:53.914777 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-config" (OuterVolumeSpecName: "console-config") pod "9bd1252a-1b63-45c5-8689-85fa8de3b532" (UID: "9bd1252a-1b63-45c5-8689-85fa8de3b532"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:53.914829 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9bd1252a-1b63-45c5-8689-85fa8de3b532" (UID: "9bd1252a-1b63-45c5-8689-85fa8de3b532"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:53.914890 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.914866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-service-ca" (OuterVolumeSpecName: "service-ca") pod "9bd1252a-1b63-45c5-8689-85fa8de3b532" (UID: "9bd1252a-1b63-45c5-8689-85fa8de3b532"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:53.916916 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.916888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9bd1252a-1b63-45c5-8689-85fa8de3b532" (UID: "9bd1252a-1b63-45c5-8689-85fa8de3b532"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:53.916916 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.916901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd1252a-1b63-45c5-8689-85fa8de3b532-kube-api-access-kjbxv" (OuterVolumeSpecName: "kube-api-access-kjbxv") pod "9bd1252a-1b63-45c5-8689-85fa8de3b532" (UID: "9bd1252a-1b63-45c5-8689-85fa8de3b532"). InnerVolumeSpecName "kube-api-access-kjbxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:53.917059 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:53.916934    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9bd1252a-1b63-45c5-8689-85fa8de3b532" (UID: "9bd1252a-1b63-45c5-8689-85fa8de3b532"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:54.015459 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.015437    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-service-ca\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:54.015459 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.015459    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbxv\" (UniqueName: \"kubernetes.io/projected/9bd1252a-1b63-45c5-8689-85fa8de3b532-kube-api-access-kjbxv\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:54.015597 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.015469    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:54.015597 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.015478    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-oauth-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:54.015597 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.015487    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-console-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:54.015597 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.015495    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd1252a-1b63-45c5-8689-85fa8de3b532-oauth-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:52:54.567630 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.567593    2577 generic.go:356] "Generic (PLEG): container finished" podID="9bd1252a-1b63-45c5-8689-85fa8de3b532" containerID="0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a" exitCode=0
Sep 17 20:52:54.568061 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.567637    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95d48c86-xlflc"
Sep 17 20:52:54.568061 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.567658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95d48c86-xlflc" event={"ID":"9bd1252a-1b63-45c5-8689-85fa8de3b532","Type":"ContainerDied","Data":"0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a"}
Sep 17 20:52:54.568061 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.567697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95d48c86-xlflc" event={"ID":"9bd1252a-1b63-45c5-8689-85fa8de3b532","Type":"ContainerDied","Data":"4efbb0be32b475bc881488c1289cce0fe23de8df681ddc2b9fcc23728b951a72"}
Sep 17 20:52:54.568061 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.567716    2577 scope.go:117] "RemoveContainer" containerID="0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a"
Sep 17 20:52:54.575754 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.575737    2577 scope.go:117] "RemoveContainer" containerID="0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a"
Sep 17 20:52:54.575996 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:52:54.575980    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a\": container with ID starting with 0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a not found: ID does not exist" containerID="0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a"
Sep 17 20:52:54.576048 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.576002    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a"} err="failed to get container status \"0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a\": rpc error: code = NotFound desc = could not find container \"0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a\": container with ID starting with 0948919b7e2047366f284b27176ab212ba89e477291bd5b05f45d48807083f8a not found: ID does not exist"
Sep 17 20:52:54.587044 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.587024    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b95d48c86-xlflc"]
Sep 17 20:52:54.590510 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:54.590489    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b95d48c86-xlflc"]
Sep 17 20:52:55.715827 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:52:55.715793    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bd1252a-1b63-45c5-8689-85fa8de3b532" path="/var/lib/kubelet/pods/9bd1252a-1b63-45c5-8689-85fa8de3b532/volumes"
Sep 17 20:53:21.527290 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.527256    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7894dd5776-4w77q"]
Sep 17 20:53:21.527740 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.527519    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd1252a-1b63-45c5-8689-85fa8de3b532" containerName="console"
Sep 17 20:53:21.527740 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.527534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd1252a-1b63-45c5-8689-85fa8de3b532" containerName="console"
Sep 17 20:53:21.527740 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.527575    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd1252a-1b63-45c5-8689-85fa8de3b532" containerName="console"
Sep 17 20:53:21.530342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.530326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.541449 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.541428    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7894dd5776-4w77q"]
Sep 17 20:53:21.623026 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.623002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-trusted-ca-bundle\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.623131 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.623047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-serving-cert\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.623131 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.623104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-console-config\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.623263 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.623142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-oauth-config\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.623263 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.623170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4bw\" (UniqueName: \"kubernetes.io/projected/2734dc4e-413d-41b9-9881-5c0047b89b72-kube-api-access-np4bw\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.623392 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.623273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-service-ca\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.623392 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.623312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-oauth-serving-cert\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.723662 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.723631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-serving-cert\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.723811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.723692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-console-config\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.723811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.723718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-oauth-config\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.723811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.723740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4bw\" (UniqueName: \"kubernetes.io/projected/2734dc4e-413d-41b9-9881-5c0047b89b72-kube-api-access-np4bw\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.723811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.723793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-service-ca\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.723978 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.723834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-oauth-serving-cert\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.723978 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.723869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-trusted-ca-bundle\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.724702 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.724673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-console-config\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.724702 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.724697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-trusted-ca-bundle\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.724875 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.724778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-service-ca\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.724956 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.724930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-oauth-serving-cert\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.726522 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.726501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-oauth-config\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.726629 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.726506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-serving-cert\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.733129 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.733107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4bw\" (UniqueName: \"kubernetes.io/projected/2734dc4e-413d-41b9-9881-5c0047b89b72-kube-api-access-np4bw\") pod \"console-7894dd5776-4w77q\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") " pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:21.838590 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:21.838502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:22.111785 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:22.111702    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7894dd5776-4w77q"]
Sep 17 20:53:22.115283 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:53:22.115232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2734dc4e_413d_41b9_9881_5c0047b89b72.slice/crio-20c25fa353f403e0d5b7b65bda816223c9e77c4f00853e9fd7f983ec1cf155cb WatchSource:0}: Error finding container 20c25fa353f403e0d5b7b65bda816223c9e77c4f00853e9fd7f983ec1cf155cb: Status 404 returned error can't find the container with id 20c25fa353f403e0d5b7b65bda816223c9e77c4f00853e9fd7f983ec1cf155cb
Sep 17 20:53:22.665412 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:22.665378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7894dd5776-4w77q" event={"ID":"2734dc4e-413d-41b9-9881-5c0047b89b72","Type":"ContainerStarted","Data":"675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384"}
Sep 17 20:53:22.665872 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:22.665469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7894dd5776-4w77q" event={"ID":"2734dc4e-413d-41b9-9881-5c0047b89b72","Type":"ContainerStarted","Data":"20c25fa353f403e0d5b7b65bda816223c9e77c4f00853e9fd7f983ec1cf155cb"}
Sep 17 20:53:22.684591 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:22.684540    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7894dd5776-4w77q" podStartSLOduration=1.684523785 podStartE2EDuration="1.684523785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:22.68443919 +0000 UTC m=+241.503731153" watchObservedRunningTime="2026-09-17 20:53:22.684523785 +0000 UTC m=+241.503815744"
Sep 17 20:53:31.839277 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:31.839219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:31.839755 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:31.839299    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:31.843865 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:31.843847    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:32.602698 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:32.602661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:53:32.605040 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:32.605023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b-metrics-certs\") pod \"network-metrics-daemon-4tl6x\" (UID: \"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b\") " pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:53:32.615711 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:32.615689    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j64n\""
Sep 17 20:53:32.623818 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:32.623800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tl6x"
Sep 17 20:53:32.702617 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:32.702593    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:53:32.752492 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:32.752463    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f588dbff9-9ztdp"]
Sep 17 20:53:32.894930 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:32.894858    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tl6x"]
Sep 17 20:53:32.897804 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:53:32.897769    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6dbb90_da4d_4e9f_b3f0_c24624e0e08b.slice/crio-85b139d64be418473d3ba41837182571f081cad14599d26aa9a61fba320b46ea WatchSource:0}: Error finding container 85b139d64be418473d3ba41837182571f081cad14599d26aa9a61fba320b46ea: Status 404 returned error can't find the container with id 85b139d64be418473d3ba41837182571f081cad14599d26aa9a61fba320b46ea
Sep 17 20:53:33.704465 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:33.704430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tl6x" event={"ID":"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b","Type":"ContainerStarted","Data":"85b139d64be418473d3ba41837182571f081cad14599d26aa9a61fba320b46ea"}
Sep 17 20:53:34.709394 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:34.709351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tl6x" event={"ID":"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b","Type":"ContainerStarted","Data":"fde46c92e9098b11b6d69beb2b88144d4b6348df6948ceba948c9d0b871b1740"}
Sep 17 20:53:34.709767 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:34.709419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tl6x" event={"ID":"2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b","Type":"ContainerStarted","Data":"80c044a0910934d868dc6fb4fcd74511f08631a3740fda67e2bdb1be03df67ca"}
Sep 17 20:53:34.728827 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:34.728775    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4tl6x" podStartSLOduration=252.865024867 podStartE2EDuration="4m13.728757685s" totalImagesPullingTime="863.732818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:34.727974456 +0000 UTC m=+253.547266430" watchObservedRunningTime="2026-09-17 20:53:34.728757685 +0000 UTC m=+253.548049664"
Sep 17 20:53:57.777091 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:57.777051    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f588dbff9-9ztdp" podUID="2ce5dffe-fb11-429c-b945-f83041733d69" containerName="console" containerID="cri-o://e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e" gracePeriod=15
Sep 17 20:53:58.023961 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.023937    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:53:58.077493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-oauth-config\") pod \"2ce5dffe-fb11-429c-b945-f83041733d69\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") "
Sep 17 20:53:58.077493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-serving-cert\") pod \"2ce5dffe-fb11-429c-b945-f83041733d69\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") "
Sep 17 20:53:58.077493 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cll7d\" (UniqueName: \"kubernetes.io/projected/2ce5dffe-fb11-429c-b945-f83041733d69-kube-api-access-cll7d\") pod \"2ce5dffe-fb11-429c-b945-f83041733d69\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") "
Sep 17 20:53:58.077714 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-trusted-ca-bundle\") pod \"2ce5dffe-fb11-429c-b945-f83041733d69\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") "
Sep 17 20:53:58.077714 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077658    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-service-ca\") pod \"2ce5dffe-fb11-429c-b945-f83041733d69\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") "
Sep 17 20:53:58.077807 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-oauth-serving-cert\") pod \"2ce5dffe-fb11-429c-b945-f83041733d69\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") "
Sep 17 20:53:58.077807 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-console-config\") pod \"2ce5dffe-fb11-429c-b945-f83041733d69\" (UID: \"2ce5dffe-fb11-429c-b945-f83041733d69\") "
Sep 17 20:53:58.078030 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.078001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-service-ca" (OuterVolumeSpecName: "service-ca") pod "2ce5dffe-fb11-429c-b945-f83041733d69" (UID: "2ce5dffe-fb11-429c-b945-f83041733d69"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:58.078030 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.077933    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2ce5dffe-fb11-429c-b945-f83041733d69" (UID: "2ce5dffe-fb11-429c-b945-f83041733d69"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:58.078155 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.078085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2ce5dffe-fb11-429c-b945-f83041733d69" (UID: "2ce5dffe-fb11-429c-b945-f83041733d69"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:58.078314 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.078296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-console-config" (OuterVolumeSpecName: "console-config") pod "2ce5dffe-fb11-429c-b945-f83041733d69" (UID: "2ce5dffe-fb11-429c-b945-f83041733d69"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:58.079643 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.079617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2ce5dffe-fb11-429c-b945-f83041733d69" (UID: "2ce5dffe-fb11-429c-b945-f83041733d69"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:58.080266 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.080221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce5dffe-fb11-429c-b945-f83041733d69-kube-api-access-cll7d" (OuterVolumeSpecName: "kube-api-access-cll7d") pod "2ce5dffe-fb11-429c-b945-f83041733d69" (UID: "2ce5dffe-fb11-429c-b945-f83041733d69"). InnerVolumeSpecName "kube-api-access-cll7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:58.080266 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.080229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2ce5dffe-fb11-429c-b945-f83041733d69" (UID: "2ce5dffe-fb11-429c-b945-f83041733d69"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:58.178582 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.178560    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-oauth-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:53:58.178582 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.178579    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-console-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:53:58.178712 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.178588    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-oauth-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:53:58.178712 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.178596    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce5dffe-fb11-429c-b945-f83041733d69-console-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:53:58.178712 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.178605    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cll7d\" (UniqueName: \"kubernetes.io/projected/2ce5dffe-fb11-429c-b945-f83041733d69-kube-api-access-cll7d\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:53:58.178712 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.178613    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-trusted-ca-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:53:58.178712 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.178621    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce5dffe-fb11-429c-b945-f83041733d69-service-ca\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:53:58.786472 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.786432    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ce5dffe-fb11-429c-b945-f83041733d69" containerID="e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e" exitCode=0
Sep 17 20:53:58.786472 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.786472    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f588dbff9-9ztdp"
Sep 17 20:53:58.786943 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.786508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f588dbff9-9ztdp" event={"ID":"2ce5dffe-fb11-429c-b945-f83041733d69","Type":"ContainerDied","Data":"e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e"}
Sep 17 20:53:58.786943 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.786548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f588dbff9-9ztdp" event={"ID":"2ce5dffe-fb11-429c-b945-f83041733d69","Type":"ContainerDied","Data":"24a2c13e18dcf396ddcff9427f454feae159a9e03685da48db733365c5a13377"}
Sep 17 20:53:58.786943 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.786568    2577 scope.go:117] "RemoveContainer" containerID="e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e"
Sep 17 20:53:58.794780 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.794759    2577 scope.go:117] "RemoveContainer" containerID="e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e"
Sep 17 20:53:58.795066 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:53:58.795045    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e\": container with ID starting with e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e not found: ID does not exist" containerID="e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e"
Sep 17 20:53:58.795134 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.795072    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e"} err="failed to get container status \"e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e\": rpc error: code = NotFound desc = could not find container \"e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e\": container with ID starting with e3a4820bbbf1fade300142034b3f9c7f094d4240897bb4ac96e34547a2808c1e not found: ID does not exist"
Sep 17 20:53:58.806393 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.806373    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f588dbff9-9ztdp"]
Sep 17 20:53:58.809823 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:58.809804    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f588dbff9-9ztdp"]
Sep 17 20:53:59.720005 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:53:59.717597    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce5dffe-fb11-429c-b945-f83041733d69" path="/var/lib/kubelet/pods/2ce5dffe-fb11-429c-b945-f83041733d69/volumes"
Sep 17 20:54:21.610140 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:21.610105    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 20:54:28.494832 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.494800    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b4f8f674f-kj2l2"]
Sep 17 20:54:28.497278 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.495128    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce5dffe-fb11-429c-b945-f83041733d69" containerName="console"
Sep 17 20:54:28.497278 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.495146    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce5dffe-fb11-429c-b945-f83041733d69" containerName="console"
Sep 17 20:54:28.497278 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.495211    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce5dffe-fb11-429c-b945-f83041733d69" containerName="console"
Sep 17 20:54:28.498119 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.498102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.517156 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.517125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b4f8f674f-kj2l2"]
Sep 17 20:54:28.584146 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.584105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-oauth-serving-cert\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.584146 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.584150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-config\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.584370 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.584175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-serving-cert\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.584370 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.584205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb67z\" (UniqueName: \"kubernetes.io/projected/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-kube-api-access-zb67z\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.584370 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.584222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-service-ca\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.584370 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.584258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-trusted-ca-bundle\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.584370 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.584340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-oauth-config\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.684981 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.684935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-oauth-config\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685169 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.684989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-oauth-serving-cert\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685169 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-config\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685169 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-serving-cert\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685169 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb67z\" (UniqueName: \"kubernetes.io/projected/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-kube-api-access-zb67z\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685169 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-service-ca\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685169 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-trusted-ca-bundle\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685918 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-oauth-serving-cert\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.685918 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-config\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.686061 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.685931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-service-ca\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.686061 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.686040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-trusted-ca-bundle\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.687619 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.687600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-oauth-config\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.687712 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.687694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-serving-cert\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.693841 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.693821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb67z\" (UniqueName: \"kubernetes.io/projected/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-kube-api-access-zb67z\") pod \"console-7b4f8f674f-kj2l2\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") " pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:28.807202 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:28.807156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:29.081512 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:29.081408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b4f8f674f-kj2l2"]
Sep 17 20:54:29.084677 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:54:29.084650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5751ab_6a44_4b02_b3b1_7b5906ad6abf.slice/crio-d6e98acc8fc151af8d44e0c7d830cfbbc33912990ffcea6b6dadb991e4d4ff6d WatchSource:0}: Error finding container d6e98acc8fc151af8d44e0c7d830cfbbc33912990ffcea6b6dadb991e4d4ff6d: Status 404 returned error can't find the container with id d6e98acc8fc151af8d44e0c7d830cfbbc33912990ffcea6b6dadb991e4d4ff6d
Sep 17 20:54:29.885103 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:29.885070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b4f8f674f-kj2l2" event={"ID":"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf","Type":"ContainerStarted","Data":"5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8"}
Sep 17 20:54:29.885498 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:29.885117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b4f8f674f-kj2l2" event={"ID":"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf","Type":"ContainerStarted","Data":"d6e98acc8fc151af8d44e0c7d830cfbbc33912990ffcea6b6dadb991e4d4ff6d"}
Sep 17 20:54:29.905225 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:29.905176    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b4f8f674f-kj2l2" podStartSLOduration=1.90516064 podStartE2EDuration="1.90516064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:54:29.904293683 +0000 UTC m=+308.723585646" watchObservedRunningTime="2026-09-17 20:54:29.90516064 +0000 UTC m=+308.724452601"
Sep 17 20:54:38.808099 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:38.808050    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:38.808099 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:38.808118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:38.812794 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:38.812770    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:38.918329 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:38.918300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:54:38.969530 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:54:38.969501    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7894dd5776-4w77q"]
Sep 17 20:55:03.994129 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:03.994055    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7894dd5776-4w77q" podUID="2734dc4e-413d-41b9-9881-5c0047b89b72" containerName="console" containerID="cri-o://675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384" gracePeriod=15
Sep 17 20:55:04.235068 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.235041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:55:04.352842 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.352810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-service-ca\") pod \"2734dc4e-413d-41b9-9881-5c0047b89b72\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") "
Sep 17 20:55:04.352842 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.352848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-trusted-ca-bundle\") pod \"2734dc4e-413d-41b9-9881-5c0047b89b72\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") "
Sep 17 20:55:04.353100 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.352873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-console-config\") pod \"2734dc4e-413d-41b9-9881-5c0047b89b72\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") "
Sep 17 20:55:04.353100 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.352902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-serving-cert\") pod \"2734dc4e-413d-41b9-9881-5c0047b89b72\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") "
Sep 17 20:55:04.353100 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.352924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-oauth-config\") pod \"2734dc4e-413d-41b9-9881-5c0047b89b72\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") "
Sep 17 20:55:04.353100 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.352965    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-oauth-serving-cert\") pod \"2734dc4e-413d-41b9-9881-5c0047b89b72\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") "
Sep 17 20:55:04.353100 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.353002    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np4bw\" (UniqueName: \"kubernetes.io/projected/2734dc4e-413d-41b9-9881-5c0047b89b72-kube-api-access-np4bw\") pod \"2734dc4e-413d-41b9-9881-5c0047b89b72\" (UID: \"2734dc4e-413d-41b9-9881-5c0047b89b72\") "
Sep 17 20:55:04.353554 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.353512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-service-ca" (OuterVolumeSpecName: "service-ca") pod "2734dc4e-413d-41b9-9881-5c0047b89b72" (UID: "2734dc4e-413d-41b9-9881-5c0047b89b72"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:04.353724 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.353572    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-console-config" (OuterVolumeSpecName: "console-config") pod "2734dc4e-413d-41b9-9881-5c0047b89b72" (UID: "2734dc4e-413d-41b9-9881-5c0047b89b72"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:04.353724 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.353617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2734dc4e-413d-41b9-9881-5c0047b89b72" (UID: "2734dc4e-413d-41b9-9881-5c0047b89b72"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:04.353841 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.353734    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2734dc4e-413d-41b9-9881-5c0047b89b72" (UID: "2734dc4e-413d-41b9-9881-5c0047b89b72"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:04.356291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.356202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2734dc4e-413d-41b9-9881-5c0047b89b72-kube-api-access-np4bw" (OuterVolumeSpecName: "kube-api-access-np4bw") pod "2734dc4e-413d-41b9-9881-5c0047b89b72" (UID: "2734dc4e-413d-41b9-9881-5c0047b89b72"). InnerVolumeSpecName "kube-api-access-np4bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:55:04.356291 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.356223    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2734dc4e-413d-41b9-9881-5c0047b89b72" (UID: "2734dc4e-413d-41b9-9881-5c0047b89b72"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:55:04.358913 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.357365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2734dc4e-413d-41b9-9881-5c0047b89b72" (UID: "2734dc4e-413d-41b9-9881-5c0047b89b72"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:55:04.453895 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.453857    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-oauth-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:55:04.453895 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.453888    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4bw\" (UniqueName: \"kubernetes.io/projected/2734dc4e-413d-41b9-9881-5c0047b89b72-kube-api-access-np4bw\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:55:04.453895 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.453899    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-service-ca\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:55:04.454109 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.453907    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-trusted-ca-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:55:04.454109 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.453916    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2734dc4e-413d-41b9-9881-5c0047b89b72-console-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:55:04.454109 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.453926    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:55:04.454109 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:04.453939    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2734dc4e-413d-41b9-9881-5c0047b89b72-console-oauth-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:55:05.001284 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.001231    2577 generic.go:356] "Generic (PLEG): container finished" podID="2734dc4e-413d-41b9-9881-5c0047b89b72" containerID="675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384" exitCode=0
Sep 17 20:55:05.001687 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.001298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7894dd5776-4w77q" event={"ID":"2734dc4e-413d-41b9-9881-5c0047b89b72","Type":"ContainerDied","Data":"675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384"}
Sep 17 20:55:05.001687 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.001306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7894dd5776-4w77q"
Sep 17 20:55:05.001687 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.001339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7894dd5776-4w77q" event={"ID":"2734dc4e-413d-41b9-9881-5c0047b89b72","Type":"ContainerDied","Data":"20c25fa353f403e0d5b7b65bda816223c9e77c4f00853e9fd7f983ec1cf155cb"}
Sep 17 20:55:05.001687 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.001358    2577 scope.go:117] "RemoveContainer" containerID="675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384"
Sep 17 20:55:05.009410 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.009386    2577 scope.go:117] "RemoveContainer" containerID="675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384"
Sep 17 20:55:05.009638 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:55:05.009621    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384\": container with ID starting with 675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384 not found: ID does not exist" containerID="675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384"
Sep 17 20:55:05.009691 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.009646    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384"} err="failed to get container status \"675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384\": rpc error: code = NotFound desc = could not find container \"675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384\": container with ID starting with 675d33806832685f985365fb3a91b8e4a7115323c3d23b5e42d46f566f246384 not found: ID does not exist"
Sep 17 20:55:05.021587 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.021565    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7894dd5776-4w77q"]
Sep 17 20:55:05.027435 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.027412    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7894dd5776-4w77q"]
Sep 17 20:55:05.716501 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:05.716469    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2734dc4e-413d-41b9-9881-5c0047b89b72" path="/var/lib/kubelet/pods/2734dc4e-413d-41b9-9881-5c0047b89b72/volumes"
Sep 17 20:55:46.182134 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.182093    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"]
Sep 17 20:55:46.182615 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.182346    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2734dc4e-413d-41b9-9881-5c0047b89b72" containerName="console"
Sep 17 20:55:46.182615 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.182358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2734dc4e-413d-41b9-9881-5c0047b89b72" containerName="console"
Sep 17 20:55:46.182615 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.182397    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2734dc4e-413d-41b9-9881-5c0047b89b72" containerName="console"
Sep 17 20:55:46.185120 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.185104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.187561 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.187539    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 20:55:46.188901 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.188076    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p6qnm\""
Sep 17 20:55:46.188901 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.188435    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 20:55:46.195825 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.195803    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"]
Sep 17 20:55:46.342500 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.342458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4xb\" (UniqueName: \"kubernetes.io/projected/f2f50549-5529-4359-9569-075ea8801749-kube-api-access-qv4xb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.342684 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.342519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.342684 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.342541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.443589 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.443508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4xb\" (UniqueName: \"kubernetes.io/projected/f2f50549-5529-4359-9569-075ea8801749-kube-api-access-qv4xb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.443589 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.443579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.443771 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.443610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.443942 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.443919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.444012 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.443969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.452223 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.452194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4xb\" (UniqueName: \"kubernetes.io/projected/f2f50549-5529-4359-9569-075ea8801749-kube-api-access-qv4xb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.498022 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.497991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:55:46.765883 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.765854    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"]
Sep 17 20:55:46.768888 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:55:46.768855    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f50549_5529_4359_9569_075ea8801749.slice/crio-83a7c7822c3c61f591e3703234261a36c947f80e94cbe88d708425f4a5affe6d WatchSource:0}: Error finding container 83a7c7822c3c61f591e3703234261a36c947f80e94cbe88d708425f4a5affe6d: Status 404 returned error can't find the container with id 83a7c7822c3c61f591e3703234261a36c947f80e94cbe88d708425f4a5affe6d
Sep 17 20:55:46.771141 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:46.771124    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:55:47.134530 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:47.134444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8" event={"ID":"f2f50549-5529-4359-9569-075ea8801749","Type":"ContainerStarted","Data":"83a7c7822c3c61f591e3703234261a36c947f80e94cbe88d708425f4a5affe6d"}
Sep 17 20:55:54.161341 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:54.161298    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2f50549-5529-4359-9569-075ea8801749" containerID="24fce6642d12db4e35683a0c9a7858deebcd77c8dfb83d5e0874eaec802c108d" exitCode=0
Sep 17 20:55:54.161746 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:54.161371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8" event={"ID":"f2f50549-5529-4359-9569-075ea8801749","Type":"ContainerDied","Data":"24fce6642d12db4e35683a0c9a7858deebcd77c8dfb83d5e0874eaec802c108d"}
Sep 17 20:55:56.171108 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:56.171079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8" event={"ID":"f2f50549-5529-4359-9569-075ea8801749","Type":"ContainerStarted","Data":"fc3947dde6037164775a312cba0252e38098aee5d5b0a9d1c84d970dd87a88f6"}
Sep 17 20:55:57.176394 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:57.176354    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2f50549-5529-4359-9569-075ea8801749" containerID="fc3947dde6037164775a312cba0252e38098aee5d5b0a9d1c84d970dd87a88f6" exitCode=0
Sep 17 20:55:57.176759 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:55:57.176400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8" event={"ID":"f2f50549-5529-4359-9569-075ea8801749","Type":"ContainerDied","Data":"fc3947dde6037164775a312cba0252e38098aee5d5b0a9d1c84d970dd87a88f6"}
Sep 17 20:56:04.203630 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:04.203588    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2f50549-5529-4359-9569-075ea8801749" containerID="9a6bbd0691d9223c14a62b2487e45ea55df0c3f7282d39725acca83543f905ec" exitCode=0
Sep 17 20:56:04.204042 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:04.203644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8" event={"ID":"f2f50549-5529-4359-9569-075ea8801749","Type":"ContainerDied","Data":"9a6bbd0691d9223c14a62b2487e45ea55df0c3f7282d39725acca83543f905ec"}
Sep 17 20:56:05.323167 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.323144    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:56:05.375033 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.374992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv4xb\" (UniqueName: \"kubernetes.io/projected/f2f50549-5529-4359-9569-075ea8801749-kube-api-access-qv4xb\") pod \"f2f50549-5529-4359-9569-075ea8801749\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") "
Sep 17 20:56:05.375033 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.375033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-util\") pod \"f2f50549-5529-4359-9569-075ea8801749\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") "
Sep 17 20:56:05.375228 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.375068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-bundle\") pod \"f2f50549-5529-4359-9569-075ea8801749\" (UID: \"f2f50549-5529-4359-9569-075ea8801749\") "
Sep 17 20:56:05.375870 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.375844    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-bundle" (OuterVolumeSpecName: "bundle") pod "f2f50549-5529-4359-9569-075ea8801749" (UID: "f2f50549-5529-4359-9569-075ea8801749"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:56:05.377320 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.377298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f50549-5529-4359-9569-075ea8801749-kube-api-access-qv4xb" (OuterVolumeSpecName: "kube-api-access-qv4xb") pod "f2f50549-5529-4359-9569-075ea8801749" (UID: "f2f50549-5529-4359-9569-075ea8801749"). InnerVolumeSpecName "kube-api-access-qv4xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:56:05.379987 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.379968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-util" (OuterVolumeSpecName: "util") pod "f2f50549-5529-4359-9569-075ea8801749" (UID: "f2f50549-5529-4359-9569-075ea8801749"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:56:05.476263 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.476160    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv4xb\" (UniqueName: \"kubernetes.io/projected/f2f50549-5529-4359-9569-075ea8801749-kube-api-access-qv4xb\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:56:05.476263 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.476194    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-util\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:56:05.476263 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:05.476206    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f50549-5529-4359-9569-075ea8801749-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:56:06.212778 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:06.212738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8" event={"ID":"f2f50549-5529-4359-9569-075ea8801749","Type":"ContainerDied","Data":"83a7c7822c3c61f591e3703234261a36c947f80e94cbe88d708425f4a5affe6d"}
Sep 17 20:56:06.212778 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:06.212764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1axzkc8"
Sep 17 20:56:06.212778 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:06.212784    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83a7c7822c3c61f591e3703234261a36c947f80e94cbe88d708425f4a5affe6d"
Sep 17 20:56:13.784507 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784476    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zhwdk"]
Sep 17 20:56:13.784958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784695    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f50549-5529-4359-9569-075ea8801749" containerName="pull"
Sep 17 20:56:13.784958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784704    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f50549-5529-4359-9569-075ea8801749" containerName="pull"
Sep 17 20:56:13.784958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784718    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f50549-5529-4359-9569-075ea8801749" containerName="extract"
Sep 17 20:56:13.784958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784723    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f50549-5529-4359-9569-075ea8801749" containerName="extract"
Sep 17 20:56:13.784958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784732    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f50549-5529-4359-9569-075ea8801749" containerName="util"
Sep 17 20:56:13.784958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784738    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f50549-5529-4359-9569-075ea8801749" containerName="util"
Sep 17 20:56:13.784958 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.784772    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f50549-5529-4359-9569-075ea8801749" containerName="extract"
Sep 17 20:56:13.787347 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.787331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:13.789444 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.789422    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 20:56:13.789955 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.789939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 20:56:13.790537 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.790521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ld964\""
Sep 17 20:56:13.790619 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.790523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 20:56:13.790619 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.790524    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 20:56:13.796438 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.796416    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zhwdk"]
Sep 17 20:56:13.843402 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.843374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34a7e6ba-9ea9-481e-ae0c-6a99c79e2207-certificates\") pod \"keda-admission-6c4d864b86-zhwdk\" (UID: \"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207\") " pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:13.843527 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.843442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnpt\" (UniqueName: \"kubernetes.io/projected/34a7e6ba-9ea9-481e-ae0c-6a99c79e2207-kube-api-access-flnpt\") pod \"keda-admission-6c4d864b86-zhwdk\" (UID: \"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207\") " pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:13.944071 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.944041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnpt\" (UniqueName: \"kubernetes.io/projected/34a7e6ba-9ea9-481e-ae0c-6a99c79e2207-kube-api-access-flnpt\") pod \"keda-admission-6c4d864b86-zhwdk\" (UID: \"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207\") " pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:13.944212 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.944078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34a7e6ba-9ea9-481e-ae0c-6a99c79e2207-certificates\") pod \"keda-admission-6c4d864b86-zhwdk\" (UID: \"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207\") " pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:13.946658 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.946638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34a7e6ba-9ea9-481e-ae0c-6a99c79e2207-certificates\") pod \"keda-admission-6c4d864b86-zhwdk\" (UID: \"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207\") " pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:13.958503 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:13.958478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnpt\" (UniqueName: \"kubernetes.io/projected/34a7e6ba-9ea9-481e-ae0c-6a99c79e2207-kube-api-access-flnpt\") pod \"keda-admission-6c4d864b86-zhwdk\" (UID: \"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207\") " pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:14.097943 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:14.097856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:14.375972 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:14.375791    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zhwdk"]
Sep 17 20:56:14.378588 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:56:14.378559    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a7e6ba_9ea9_481e_ae0c_6a99c79e2207.slice/crio-971cb044495aad695c516b00277e3aefca6209270adfa00fecbc2f4ff90472c9 WatchSource:0}: Error finding container 971cb044495aad695c516b00277e3aefca6209270adfa00fecbc2f4ff90472c9: Status 404 returned error can't find the container with id 971cb044495aad695c516b00277e3aefca6209270adfa00fecbc2f4ff90472c9
Sep 17 20:56:15.245106 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:15.245070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zhwdk" event={"ID":"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207","Type":"ContainerStarted","Data":"971cb044495aad695c516b00277e3aefca6209270adfa00fecbc2f4ff90472c9"}
Sep 17 20:56:16.249916 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:16.249824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zhwdk" event={"ID":"34a7e6ba-9ea9-481e-ae0c-6a99c79e2207","Type":"ContainerStarted","Data":"531c1c2f78e887223b3e4609723bf66973b66adec4a45859c8aa0f868133f4d5"}
Sep 17 20:56:16.250407 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:16.249955    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:56:16.266920 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:16.266869    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-zhwdk" podStartSLOduration=1.818172959 podStartE2EDuration="3.266853816s" totalImagesPullingTime="1.448680857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:16.266230045 +0000 UTC m=+415.085522008" watchObservedRunningTime="2026-09-17 20:56:16.266853816 +0000 UTC m=+415.086145777"
Sep 17 20:56:37.255352 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:56:37.255322    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-zhwdk"
Sep 17 20:57:07.410109 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.410076    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"]
Sep 17 20:57:07.413620 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.413604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.416061 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.416041    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p6qnm\""
Sep 17 20:57:07.416184 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.416131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 20:57:07.416812 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.416795    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 20:57:07.422074 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.422054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"]
Sep 17 20:57:07.519779 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.519758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.519899 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.519790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvk8\" (UniqueName: \"kubernetes.io/projected/a381b2f5-5d27-48b9-9593-e7116aaf376c-kube-api-access-sqvk8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.519899 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.519822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.620337 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.620306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.620473 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.620378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.620473 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.620410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvk8\" (UniqueName: \"kubernetes.io/projected/a381b2f5-5d27-48b9-9593-e7116aaf376c-kube-api-access-sqvk8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.620661 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.620643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.620707 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.620665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.629431 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.629406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvk8\" (UniqueName: \"kubernetes.io/projected/a381b2f5-5d27-48b9-9593-e7116aaf376c-kube-api-access-sqvk8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.723141 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.723081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:07.992821 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:07.992803    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"]
Sep 17 20:57:07.995559 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:57:07.995527    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda381b2f5_5d27_48b9_9593_e7116aaf376c.slice/crio-a7b28d3a93a3a413ec938cfd24c76706cc974b48c6802ce4b1cdf191c3424d9d WatchSource:0}: Error finding container a7b28d3a93a3a413ec938cfd24c76706cc974b48c6802ce4b1cdf191c3424d9d: Status 404 returned error can't find the container with id a7b28d3a93a3a413ec938cfd24c76706cc974b48c6802ce4b1cdf191c3424d9d
Sep 17 20:57:08.422533 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:08.422498    2577 generic.go:356] "Generic (PLEG): container finished" podID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerID="86e7e761902494d72d16b3d8b4a1960471ecd793fb79e0fc9e246edc9796b525" exitCode=0
Sep 17 20:57:08.422894 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:08.422569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2" event={"ID":"a381b2f5-5d27-48b9-9593-e7116aaf376c","Type":"ContainerDied","Data":"86e7e761902494d72d16b3d8b4a1960471ecd793fb79e0fc9e246edc9796b525"}
Sep 17 20:57:08.422894 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:08.422620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2" event={"ID":"a381b2f5-5d27-48b9-9593-e7116aaf376c","Type":"ContainerStarted","Data":"a7b28d3a93a3a413ec938cfd24c76706cc974b48c6802ce4b1cdf191c3424d9d"}
Sep 17 20:57:11.435707 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.435669    2577 generic.go:356] "Generic (PLEG): container finished" podID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerID="2197fcd8f792f97fb73d468f98ff7ee546f5d740da6a37ecf0b7ed767e661a24" exitCode=0
Sep 17 20:57:11.436076 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.435749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2" event={"ID":"a381b2f5-5d27-48b9-9593-e7116aaf376c","Type":"ContainerDied","Data":"2197fcd8f792f97fb73d468f98ff7ee546f5d740da6a37ecf0b7ed767e661a24"}
Sep 17 20:57:11.729600 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.729524    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cff44857c-q59vq"]
Sep 17 20:57:11.732419 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.732404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.744091 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.744062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cff44857c-q59vq"]
Sep 17 20:57:11.850386 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.850353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f467411-4bd4-457f-823f-467c9b019925-console-oauth-config\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.850570 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.850396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f467411-4bd4-457f-823f-467c9b019925-console-serving-cert\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.850570 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.850425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-console-config\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.850570 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.850449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88lgc\" (UniqueName: \"kubernetes.io/projected/1f467411-4bd4-457f-823f-467c9b019925-kube-api-access-88lgc\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.850570 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.850518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-service-ca\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.850570 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.850540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-trusted-ca-bundle\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.850570 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.850560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-oauth-serving-cert\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.951779 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.951743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f467411-4bd4-457f-823f-467c9b019925-console-oauth-config\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.951779 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.951782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f467411-4bd4-457f-823f-467c9b019925-console-serving-cert\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.951989 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.951809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-console-config\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.951989 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.951833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88lgc\" (UniqueName: \"kubernetes.io/projected/1f467411-4bd4-457f-823f-467c9b019925-kube-api-access-88lgc\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.951989 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.951874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-service-ca\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.951989 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.951895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-trusted-ca-bundle\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.951989 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.951916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-oauth-serving-cert\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.952573 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.952550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-oauth-serving-cert\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.952674 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.952599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-console-config\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.952775 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.952756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-service-ca\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.953188 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.953163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f467411-4bd4-457f-823f-467c9b019925-trusted-ca-bundle\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.954377 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.954357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f467411-4bd4-457f-823f-467c9b019925-console-serving-cert\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.954512 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.954492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f467411-4bd4-457f-823f-467c9b019925-console-oauth-config\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:11.960265 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:11.960227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88lgc\" (UniqueName: \"kubernetes.io/projected/1f467411-4bd4-457f-823f-467c9b019925-kube-api-access-88lgc\") pod \"console-6cff44857c-q59vq\" (UID: \"1f467411-4bd4-457f-823f-467c9b019925\") " pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:12.041097 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:12.041063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:12.314048 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:12.313925    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cff44857c-q59vq"]
Sep 17 20:57:12.316515 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:57:12.316488    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f467411_4bd4_457f_823f_467c9b019925.slice/crio-c14371243b957f4b0f49c6ba0192e92ab12434ca6a8c4926411c95cef68b10c9 WatchSource:0}: Error finding container c14371243b957f4b0f49c6ba0192e92ab12434ca6a8c4926411c95cef68b10c9: Status 404 returned error can't find the container with id c14371243b957f4b0f49c6ba0192e92ab12434ca6a8c4926411c95cef68b10c9
Sep 17 20:57:12.440839 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:12.440805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cff44857c-q59vq" event={"ID":"1f467411-4bd4-457f-823f-467c9b019925","Type":"ContainerStarted","Data":"bd9330fccc814f93e9164953dea85193841d11e7d4da8b100d00d7baad0a4fc3"}
Sep 17 20:57:12.441298 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:12.440862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cff44857c-q59vq" event={"ID":"1f467411-4bd4-457f-823f-467c9b019925","Type":"ContainerStarted","Data":"c14371243b957f4b0f49c6ba0192e92ab12434ca6a8c4926411c95cef68b10c9"}
Sep 17 20:57:12.442907 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:12.442882    2577 generic.go:356] "Generic (PLEG): container finished" podID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerID="262fef1f8998a035dbbfa03bf44ec804f6e48c3647ade49c36e6a1396bb2decc" exitCode=0
Sep 17 20:57:12.443028 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:12.442946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2" event={"ID":"a381b2f5-5d27-48b9-9593-e7116aaf376c","Type":"ContainerDied","Data":"262fef1f8998a035dbbfa03bf44ec804f6e48c3647ade49c36e6a1396bb2decc"}
Sep 17 20:57:12.463292 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:12.463225    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cff44857c-q59vq" podStartSLOduration=1.463209542 podStartE2EDuration="1.463209542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:12.461959784 +0000 UTC m=+471.281251748" watchObservedRunningTime="2026-09-17 20:57:12.463209542 +0000 UTC m=+471.282501504"
Sep 17 20:57:13.568641 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.568616    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:13.664040 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.663999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-bundle\") pod \"a381b2f5-5d27-48b9-9593-e7116aaf376c\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") "
Sep 17 20:57:13.664219 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.664050    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqvk8\" (UniqueName: \"kubernetes.io/projected/a381b2f5-5d27-48b9-9593-e7116aaf376c-kube-api-access-sqvk8\") pod \"a381b2f5-5d27-48b9-9593-e7116aaf376c\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") "
Sep 17 20:57:13.664219 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.664080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-util\") pod \"a381b2f5-5d27-48b9-9593-e7116aaf376c\" (UID: \"a381b2f5-5d27-48b9-9593-e7116aaf376c\") "
Sep 17 20:57:13.664733 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.664707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-bundle" (OuterVolumeSpecName: "bundle") pod "a381b2f5-5d27-48b9-9593-e7116aaf376c" (UID: "a381b2f5-5d27-48b9-9593-e7116aaf376c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:57:13.666224 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.666202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a381b2f5-5d27-48b9-9593-e7116aaf376c-kube-api-access-sqvk8" (OuterVolumeSpecName: "kube-api-access-sqvk8") pod "a381b2f5-5d27-48b9-9593-e7116aaf376c" (UID: "a381b2f5-5d27-48b9-9593-e7116aaf376c"). InnerVolumeSpecName "kube-api-access-sqvk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:57:13.669279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.669228    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-util" (OuterVolumeSpecName: "util") pod "a381b2f5-5d27-48b9-9593-e7116aaf376c" (UID: "a381b2f5-5d27-48b9-9593-e7116aaf376c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:57:13.764988 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.764958    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:13.764988 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.764984    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvk8\" (UniqueName: \"kubernetes.io/projected/a381b2f5-5d27-48b9-9593-e7116aaf376c-kube-api-access-sqvk8\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:13.765138 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:13.764998    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a381b2f5-5d27-48b9-9593-e7116aaf376c-util\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:14.451964 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:14.451932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2" event={"ID":"a381b2f5-5d27-48b9-9593-e7116aaf376c","Type":"ContainerDied","Data":"a7b28d3a93a3a413ec938cfd24c76706cc974b48c6802ce4b1cdf191c3424d9d"}
Sep 17 20:57:14.452124 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:14.451978    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b28d3a93a3a413ec938cfd24c76706cc974b48c6802ce4b1cdf191c3424d9d"
Sep 17 20:57:14.452124 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:14.451942    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404qdn2"
Sep 17 20:57:22.041942 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:22.041898    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:22.042472 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:22.041977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:22.046645 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:22.046622    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:22.484505 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:22.484423    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cff44857c-q59vq"
Sep 17 20:57:22.594034 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:22.594004    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b4f8f674f-kj2l2"]
Sep 17 20:57:29.323838 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.323804    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"]
Sep 17 20:57:29.324356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.324024    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerName="extract"
Sep 17 20:57:29.324356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.324035    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerName="extract"
Sep 17 20:57:29.324356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.324048    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerName="util"
Sep 17 20:57:29.324356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.324053    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerName="util"
Sep 17 20:57:29.324356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.324060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerName="pull"
Sep 17 20:57:29.324356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.324066    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerName="pull"
Sep 17 20:57:29.324356 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.324106    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a381b2f5-5d27-48b9-9593-e7116aaf376c" containerName="extract"
Sep 17 20:57:29.330147 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.330128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.346497 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.346478    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 20:57:29.346607 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.346499    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 20:57:29.346674 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.346660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"]
Sep 17 20:57:29.347321 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.347304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jsjfv\""
Sep 17 20:57:29.376721 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.376687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74256144-4cb0-47e1-ba79-a4fa1a507aaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmrb6\" (UID: \"74256144-4cb0-47e1-ba79-a4fa1a507aaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.376827 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.376751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58wt\" (UniqueName: \"kubernetes.io/projected/74256144-4cb0-47e1-ba79-a4fa1a507aaa-kube-api-access-l58wt\") pod \"cert-manager-cainjector-559cdd8c87-qmrb6\" (UID: \"74256144-4cb0-47e1-ba79-a4fa1a507aaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.477323 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.477288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74256144-4cb0-47e1-ba79-a4fa1a507aaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmrb6\" (UID: \"74256144-4cb0-47e1-ba79-a4fa1a507aaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.477480 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.477358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58wt\" (UniqueName: \"kubernetes.io/projected/74256144-4cb0-47e1-ba79-a4fa1a507aaa-kube-api-access-l58wt\") pod \"cert-manager-cainjector-559cdd8c87-qmrb6\" (UID: \"74256144-4cb0-47e1-ba79-a4fa1a507aaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.507327 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.507300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74256144-4cb0-47e1-ba79-a4fa1a507aaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmrb6\" (UID: \"74256144-4cb0-47e1-ba79-a4fa1a507aaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.507910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.507888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58wt\" (UniqueName: \"kubernetes.io/projected/74256144-4cb0-47e1-ba79-a4fa1a507aaa-kube-api-access-l58wt\") pod \"cert-manager-cainjector-559cdd8c87-qmrb6\" (UID: \"74256144-4cb0-47e1-ba79-a4fa1a507aaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.639105 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.639028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"
Sep 17 20:57:29.943711 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:29.943643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6"]
Sep 17 20:57:29.948046 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:57:29.947516    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74256144_4cb0_47e1_ba79_a4fa1a507aaa.slice/crio-3a731e2ef216c5b8e5ee4532302fdf22f68b20dd0ad870de0283eb66a1fc013c WatchSource:0}: Error finding container 3a731e2ef216c5b8e5ee4532302fdf22f68b20dd0ad870de0283eb66a1fc013c: Status 404 returned error can't find the container with id 3a731e2ef216c5b8e5ee4532302fdf22f68b20dd0ad870de0283eb66a1fc013c
Sep 17 20:57:30.511906 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.511861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6" event={"ID":"74256144-4cb0-47e1-ba79-a4fa1a507aaa","Type":"ContainerStarted","Data":"3a731e2ef216c5b8e5ee4532302fdf22f68b20dd0ad870de0283eb66a1fc013c"}
Sep 17 20:57:30.928288 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.928204    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"]
Sep 17 20:57:30.932501 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.932486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:30.937290 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.937269    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 20:57:30.938164 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.938145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p6qnm\""
Sep 17 20:57:30.942450 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.942437    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 20:57:30.949466 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.949437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"]
Sep 17 20:57:30.986781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.986756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:30.986909 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.986788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:30.986909 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:30.986884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwpg\" (UniqueName: \"kubernetes.io/projected/8bd64669-604b-463b-8f64-5c2447aff685-kube-api-access-mkwpg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.087748 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.087714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.087885 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.087783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwpg\" (UniqueName: \"kubernetes.io/projected/8bd64669-604b-463b-8f64-5c2447aff685-kube-api-access-mkwpg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.087885 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.087806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.088170 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.088154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.088205 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.088152    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.101770 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.101741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwpg\" (UniqueName: \"kubernetes.io/projected/8bd64669-604b-463b-8f64-5c2447aff685-kube-api-access-mkwpg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.240766 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.240682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:31.612807 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:57:31.612778    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd64669_604b_463b_8f64_5c2447aff685.slice/crio-1007f9c05edc346982555a117e272e15488ae2599e83189e24a37dc1b3af6d7b WatchSource:0}: Error finding container 1007f9c05edc346982555a117e272e15488ae2599e83189e24a37dc1b3af6d7b: Status 404 returned error can't find the container with id 1007f9c05edc346982555a117e272e15488ae2599e83189e24a37dc1b3af6d7b
Sep 17 20:57:31.618079 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:31.618055    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"]
Sep 17 20:57:32.519776 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:32.519746    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bd64669-604b-463b-8f64-5c2447aff685" containerID="c26c54893faded237aad54d1214551104d56256ed92376c5f03e8a5fd0224deb" exitCode=0
Sep 17 20:57:32.519926 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:32.519837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7" event={"ID":"8bd64669-604b-463b-8f64-5c2447aff685","Type":"ContainerDied","Data":"c26c54893faded237aad54d1214551104d56256ed92376c5f03e8a5fd0224deb"}
Sep 17 20:57:32.519926 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:32.519885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7" event={"ID":"8bd64669-604b-463b-8f64-5c2447aff685","Type":"ContainerStarted","Data":"1007f9c05edc346982555a117e272e15488ae2599e83189e24a37dc1b3af6d7b"}
Sep 17 20:57:34.529109 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:34.529067    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bd64669-604b-463b-8f64-5c2447aff685" containerID="45453091bfbdf740a2ccfab1fb5f02cf618264c21a650e8de1f0613eb2469e14" exitCode=0
Sep 17 20:57:34.529627 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:34.529157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7" event={"ID":"8bd64669-604b-463b-8f64-5c2447aff685","Type":"ContainerDied","Data":"45453091bfbdf740a2ccfab1fb5f02cf618264c21a650e8de1f0613eb2469e14"}
Sep 17 20:57:35.535023 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:35.534987    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bd64669-604b-463b-8f64-5c2447aff685" containerID="4b6c57d21aea2d2555fecd8fd05f09243046f145aba2558165cef1ea94271eca" exitCode=0
Sep 17 20:57:35.535405 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:35.535079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7" event={"ID":"8bd64669-604b-463b-8f64-5c2447aff685","Type":"ContainerDied","Data":"4b6c57d21aea2d2555fecd8fd05f09243046f145aba2558165cef1ea94271eca"}
Sep 17 20:57:36.664113 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.664092    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:36.728322 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.728287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkwpg\" (UniqueName: \"kubernetes.io/projected/8bd64669-604b-463b-8f64-5c2447aff685-kube-api-access-mkwpg\") pod \"8bd64669-604b-463b-8f64-5c2447aff685\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") "
Sep 17 20:57:36.728476 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.728356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-util\") pod \"8bd64669-604b-463b-8f64-5c2447aff685\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") "
Sep 17 20:57:36.728476 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.728379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-bundle\") pod \"8bd64669-604b-463b-8f64-5c2447aff685\" (UID: \"8bd64669-604b-463b-8f64-5c2447aff685\") "
Sep 17 20:57:36.728769 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.728746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-bundle" (OuterVolumeSpecName: "bundle") pod "8bd64669-604b-463b-8f64-5c2447aff685" (UID: "8bd64669-604b-463b-8f64-5c2447aff685"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:57:36.730565 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.730545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd64669-604b-463b-8f64-5c2447aff685-kube-api-access-mkwpg" (OuterVolumeSpecName: "kube-api-access-mkwpg") pod "8bd64669-604b-463b-8f64-5c2447aff685" (UID: "8bd64669-604b-463b-8f64-5c2447aff685"). InnerVolumeSpecName "kube-api-access-mkwpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:57:36.733951 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.733919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-util" (OuterVolumeSpecName: "util") pod "8bd64669-604b-463b-8f64-5c2447aff685" (UID: "8bd64669-604b-463b-8f64-5c2447aff685"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:57:36.829565 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.829485    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwpg\" (UniqueName: \"kubernetes.io/projected/8bd64669-604b-463b-8f64-5c2447aff685-kube-api-access-mkwpg\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:36.829565 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.829512    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-util\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:36.829565 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:36.829524    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bd64669-604b-463b-8f64-5c2447aff685-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:37.543680 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:37.543652    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7"
Sep 17 20:57:37.543680 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:37.543650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w8f7" event={"ID":"8bd64669-604b-463b-8f64-5c2447aff685","Type":"ContainerDied","Data":"1007f9c05edc346982555a117e272e15488ae2599e83189e24a37dc1b3af6d7b"}
Sep 17 20:57:37.543872 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:37.543699    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1007f9c05edc346982555a117e272e15488ae2599e83189e24a37dc1b3af6d7b"
Sep 17 20:57:47.612739 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.612697    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b4f8f674f-kj2l2" podUID="ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" containerName="console" containerID="cri-o://5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8" gracePeriod=15
Sep 17 20:57:47.858622 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.858599    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:57:47.904335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-oauth-serving-cert\") pod \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") "
Sep 17 20:57:47.904335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904314    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-service-ca\") pod \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") "
Sep 17 20:57:47.904545 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-trusted-ca-bundle\") pod \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") "
Sep 17 20:57:47.904545 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-oauth-config\") pod \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") "
Sep 17 20:57:47.904545 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-serving-cert\") pod \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") "
Sep 17 20:57:47.904767 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" (UID: "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:47.904872 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904744    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb67z\" (UniqueName: \"kubernetes.io/projected/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-kube-api-access-zb67z\") pod \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") "
Sep 17 20:57:47.904872 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-config\") pod \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\" (UID: \"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf\") "
Sep 17 20:57:47.904964 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.904875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-service-ca" (OuterVolumeSpecName: "service-ca") pod "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" (UID: "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:47.905095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.905050    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-oauth-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:47.905095 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.905071    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-service-ca\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:47.905408 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.905297    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" (UID: "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:47.905408 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.905367    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-config" (OuterVolumeSpecName: "console-config") pod "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" (UID: "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:47.906982 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.906934    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" (UID: "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:57:47.906982 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.906968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" (UID: "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:57:47.907135 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:47.907079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-kube-api-access-zb67z" (OuterVolumeSpecName: "kube-api-access-zb67z") pod "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" (UID: "ba5751ab-6a44-4b02-b3b1-7b5906ad6abf"). InnerVolumeSpecName "kube-api-access-zb67z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:57:48.005715 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.005674    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-trusted-ca-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:48.005715 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.005706    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-oauth-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:48.005715 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.005717    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-serving-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:48.005942 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.005729    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb67z\" (UniqueName: \"kubernetes.io/projected/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-kube-api-access-zb67z\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:48.005942 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.005741    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf-console-config\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:57:48.586370 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.586330    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" containerID="5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8" exitCode=0
Sep 17 20:57:48.586558 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.586380    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b4f8f674f-kj2l2"
Sep 17 20:57:48.586558 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.586425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b4f8f674f-kj2l2" event={"ID":"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf","Type":"ContainerDied","Data":"5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8"}
Sep 17 20:57:48.586558 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.586487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b4f8f674f-kj2l2" event={"ID":"ba5751ab-6a44-4b02-b3b1-7b5906ad6abf","Type":"ContainerDied","Data":"d6e98acc8fc151af8d44e0c7d830cfbbc33912990ffcea6b6dadb991e4d4ff6d"}
Sep 17 20:57:48.586558 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.586517    2577 scope.go:117] "RemoveContainer" containerID="5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8"
Sep 17 20:57:48.588203 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.588180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6" event={"ID":"74256144-4cb0-47e1-ba79-a4fa1a507aaa","Type":"ContainerStarted","Data":"711d1b095bf30d882260aee49e1aacee671e8a0da3fd2eb1d3be03018fb20956"}
Sep 17 20:57:48.594872 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.594854    2577 scope.go:117] "RemoveContainer" containerID="5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8"
Sep 17 20:57:48.595143 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:57:48.595126    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8\": container with ID starting with 5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8 not found: ID does not exist" containerID="5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8"
Sep 17 20:57:48.595208 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.595152    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8"} err="failed to get container status \"5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8\": rpc error: code = NotFound desc = could not find container \"5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8\": container with ID starting with 5bfbccbfbb9a15dd809ddd480aa5d80ee1a5f902c9bcf2feff28873ced2119d8 not found: ID does not exist"
Sep 17 20:57:48.613653 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.613611    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmrb6" podStartSLOduration=1.911149883 podStartE2EDuration="19.613597962s" totalImagesPullingTime="17.702448079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:48.611214115 +0000 UTC m=+507.430506076" watchObservedRunningTime="2026-09-17 20:57:48.613597962 +0000 UTC m=+507.432889924"
Sep 17 20:57:48.635194 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.635170    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b4f8f674f-kj2l2"]
Sep 17 20:57:48.641620 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:48.641599    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b4f8f674f-kj2l2"]
Sep 17 20:57:49.715999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:49.715963    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" path="/var/lib/kubelet/pods/ba5751ab-6a44-4b02-b3b1-7b5906ad6abf/volumes"
Sep 17 20:57:57.776576 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776534    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"]
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776762    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd64669-604b-463b-8f64-5c2447aff685" containerName="util"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776773    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd64669-604b-463b-8f64-5c2447aff685" containerName="util"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776784    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd64669-604b-463b-8f64-5c2447aff685" containerName="pull"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776791    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd64669-604b-463b-8f64-5c2447aff685" containerName="pull"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776796    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd64669-604b-463b-8f64-5c2447aff685" containerName="extract"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776802    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd64669-604b-463b-8f64-5c2447aff685" containerName="extract"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776817    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" containerName="console"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776822    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" containerName="console"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776859    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd64669-604b-463b-8f64-5c2447aff685" containerName="extract"
Sep 17 20:57:57.777001 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.776866    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba5751ab-6a44-4b02-b3b1-7b5906ad6abf" containerName="console"
Sep 17 20:57:57.780904 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.780888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.783335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.783309    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 20:57:57.783335 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.783318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 20:57:57.783505 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.783325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 20:57:57.783505 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.783343    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-stvmk\""
Sep 17 20:57:57.783505 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.783365    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 20:57:57.783675 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.783662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 20:57:57.783722 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.783688    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 20:57:57.792678 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.792658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"]
Sep 17 20:57:57.876723 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.876691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.876723 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.876727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.876905 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.876746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.876905 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.876802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.876905 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.876862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.876905 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.876879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspvd\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-kube-api-access-sspvd\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.877033 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.876905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.978199 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.978162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.978330 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.978201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspvd\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-kube-api-access-sspvd\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.978330 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.978221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.978330 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.978281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.978330 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.978305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.978330 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.978320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.978539 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.978381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.979590 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.979565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.985089 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.981571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.985089 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.981627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.985089 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.981687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.985089 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.981745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.986838 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.986815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspvd\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-kube-api-access-sspvd\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:57.987438 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:57.987416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xn8sn\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:58.089781 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:58.089703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:57:58.365604 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:58.365577    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"]
Sep 17 20:57:58.367814 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:57:58.367781    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf924b7_9d12_4ae4_89c2_ae7d56e7fed0.slice/crio-95f27f7b2f97a89ecb9f90c7808a5bd0166af0a3d14dfc92e62027ecca6511c6 WatchSource:0}: Error finding container 95f27f7b2f97a89ecb9f90c7808a5bd0166af0a3d14dfc92e62027ecca6511c6: Status 404 returned error can't find the container with id 95f27f7b2f97a89ecb9f90c7808a5bd0166af0a3d14dfc92e62027ecca6511c6
Sep 17 20:57:58.626252 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:57:58.626153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn" event={"ID":"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0","Type":"ContainerStarted","Data":"95f27f7b2f97a89ecb9f90c7808a5bd0166af0a3d14dfc92e62027ecca6511c6"}
Sep 17 20:58:00.976425 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:00.976385    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 20:58:00.976705 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:00.976458    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 20:58:01.640684 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:01.640652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn" event={"ID":"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0","Type":"ContainerStarted","Data":"aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7"}
Sep 17 20:58:01.640856 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:01.640750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:58:01.663926 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:01.663858    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn" podStartSLOduration=2.057521411 podStartE2EDuration="4.663838818s" totalImagesPullingTime="2.606317407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:01.662619339 +0000 UTC m=+520.481911311" watchObservedRunningTime="2026-09-17 20:58:01.663838818 +0000 UTC m=+520.483130781"
Sep 17 20:58:02.645760 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:02.645727    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-xn8sn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 20:58:02.646111 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:02.645780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn" podUID="9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 20:58:05.645583 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:05.645555    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 20:58:18.419521 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.419442    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"]
Sep 17 20:58:18.422535 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.422520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.425120 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.425088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 20:58:18.425277 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.425149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p6qnm\""
Sep 17 20:58:18.426072 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.426039    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 20:58:18.432217 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.432193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"]
Sep 17 20:58:18.518162 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.518130    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"]
Sep 17 20:58:18.521381 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.521366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.529748 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.529722    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"]
Sep 17 20:58:18.535581 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.535557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.535689 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.535624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfjz\" (UniqueName: \"kubernetes.io/projected/c22e18cc-b0c8-4c08-904c-3a8717ee4795-kube-api-access-cwfjz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.535689 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.535665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.610579 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.610545    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"]
Sep 17 20:58:18.613597 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.613582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.622539 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.622511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"]
Sep 17 20:58:18.636215 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.636478 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9xs\" (UniqueName: \"kubernetes.io/projected/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-kube-api-access-hr9xs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.636478 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfjz\" (UniqueName: \"kubernetes.io/projected/c22e18cc-b0c8-4c08-904c-3a8717ee4795-kube-api-access-cwfjz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.636478 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.636478 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.636478 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.636866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.636866 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.636850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.644279 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.644260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfjz\" (UniqueName: \"kubernetes.io/projected/c22e18cc-b0c8-4c08-904c-3a8717ee4795-kube-api-access-cwfjz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.709708 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.709643    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"]
Sep 17 20:58:18.712945 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.712931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.722187 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.722167    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"]
Sep 17 20:58:18.733362 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.733342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:18.737131 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.737200 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.737200 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9xs\" (UniqueName: \"kubernetes.io/projected/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-kube-api-access-hr9xs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.737200 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.737372 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.737372 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7487\" (UniqueName: \"kubernetes.io/projected/7586f93e-91c4-4cd3-8606-4553b4b0d362-kube-api-access-m7487\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.737557 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.737595 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.737555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.746213 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.746191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9xs\" (UniqueName: \"kubernetes.io/projected/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-kube-api-access-hr9xs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.830808 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.830777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:18.837967 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.837946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.838047 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.837980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz84q\" (UniqueName: \"kubernetes.io/projected/fa91511e-02db-479b-971f-8ae2ea22c359-kube-api-access-lz84q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.838047 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.838013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.838141 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.838057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.838198 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.838167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.838268 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.838251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7487\" (UniqueName: \"kubernetes.io/projected/7586f93e-91c4-4cd3-8606-4553b4b0d362-kube-api-access-m7487\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.838456 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.838431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.838530 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.838440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.846852 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.846829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7487\" (UniqueName: \"kubernetes.io/projected/7586f93e-91c4-4cd3-8606-4553b4b0d362-kube-api-access-m7487\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.922739 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.922702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:18.939073 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.939047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz84q\" (UniqueName: \"kubernetes.io/projected/fa91511e-02db-479b-971f-8ae2ea22c359-kube-api-access-lz84q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.939179 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.939102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.939179 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.939124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.939472 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.939455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.939515 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.939498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:18.948342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:18.948316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz84q\" (UniqueName: \"kubernetes.io/projected/fa91511e-02db-479b-971f-8ae2ea22c359-kube-api-access-lz84q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:19.006278 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.006229    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"]
Sep 17 20:58:19.008847 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:58:19.008821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22e18cc_b0c8_4c08_904c_3a8717ee4795.slice/crio-9371001e53e6af9cec3acd01a2369364475e0f429539ce2ae997b5a9fd0d7b00 WatchSource:0}: Error finding container 9371001e53e6af9cec3acd01a2369364475e0f429539ce2ae997b5a9fd0d7b00: Status 404 returned error can't find the container with id 9371001e53e6af9cec3acd01a2369364475e0f429539ce2ae997b5a9fd0d7b00
Sep 17 20:58:19.021937 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.021917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:19.110937 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.110914    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"]
Sep 17 20:58:19.162023 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:58:19.161975    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3815c6b8_16bb_45d5_aa29_2fe0c62c7f64.slice/crio-c7ab56938c0c70fb66f5f186b039b01313f257d83a264d246eafe480d5ae0d97 WatchSource:0}: Error finding container c7ab56938c0c70fb66f5f186b039b01313f257d83a264d246eafe480d5ae0d97: Status 404 returned error can't find the container with id c7ab56938c0c70fb66f5f186b039b01313f257d83a264d246eafe480d5ae0d97
Sep 17 20:58:19.197183 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.197161    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"]
Sep 17 20:58:19.198859 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:58:19.198836    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7586f93e_91c4_4cd3_8606_4553b4b0d362.slice/crio-2699def52cfa8f98bc3c3bc71e5ec1c23d450c799ebef978a4933c1a06923858 WatchSource:0}: Error finding container 2699def52cfa8f98bc3c3bc71e5ec1c23d450c799ebef978a4933c1a06923858: Status 404 returned error can't find the container with id 2699def52cfa8f98bc3c3bc71e5ec1c23d450c799ebef978a4933c1a06923858
Sep 17 20:58:19.294594 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.294566    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"]
Sep 17 20:58:19.359099 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:58:19.359065    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa91511e_02db_479b_971f_8ae2ea22c359.slice/crio-b3789a89015ff0ee1685fd8a2744f34f7f3eb7f28778880a1915efe200ac218b WatchSource:0}: Error finding container b3789a89015ff0ee1685fd8a2744f34f7f3eb7f28778880a1915efe200ac218b: Status 404 returned error can't find the container with id b3789a89015ff0ee1685fd8a2744f34f7f3eb7f28778880a1915efe200ac218b
Sep 17 20:58:19.706985 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.706952    2577 generic.go:356] "Generic (PLEG): container finished" podID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerID="b33e8a0dead76be2bfaea4182afee656b5dd6822bcc16c597f01fadb85d94daa" exitCode=0
Sep 17 20:58:19.707543 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.707034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht" event={"ID":"7586f93e-91c4-4cd3-8606-4553b4b0d362","Type":"ContainerDied","Data":"b33e8a0dead76be2bfaea4182afee656b5dd6822bcc16c597f01fadb85d94daa"}
Sep 17 20:58:19.707543 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.707109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht" event={"ID":"7586f93e-91c4-4cd3-8606-4553b4b0d362","Type":"ContainerStarted","Data":"2699def52cfa8f98bc3c3bc71e5ec1c23d450c799ebef978a4933c1a06923858"}
Sep 17 20:58:19.708601 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.708583    2577 generic.go:356] "Generic (PLEG): container finished" podID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerID="b0eef795602998c051334419b99e2145fd417480f677fe443d95b04afab4b9be" exitCode=0
Sep 17 20:58:19.708717 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.708682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn" event={"ID":"c22e18cc-b0c8-4c08-904c-3a8717ee4795","Type":"ContainerDied","Data":"b0eef795602998c051334419b99e2145fd417480f677fe443d95b04afab4b9be"}
Sep 17 20:58:19.708831 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.708738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn" event={"ID":"c22e18cc-b0c8-4c08-904c-3a8717ee4795","Type":"ContainerStarted","Data":"9371001e53e6af9cec3acd01a2369364475e0f429539ce2ae997b5a9fd0d7b00"}
Sep 17 20:58:19.710141 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.710120    2577 generic.go:356] "Generic (PLEG): container finished" podID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerID="08ee98d80f9ca19153f15eba2deed5cd66ef7bedab8b3e7e1823a2d6c99ac3dd" exitCode=0
Sep 17 20:58:19.710258 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.710200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695" event={"ID":"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64","Type":"ContainerDied","Data":"08ee98d80f9ca19153f15eba2deed5cd66ef7bedab8b3e7e1823a2d6c99ac3dd"}
Sep 17 20:58:19.710312 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.710259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695" event={"ID":"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64","Type":"ContainerStarted","Data":"c7ab56938c0c70fb66f5f186b039b01313f257d83a264d246eafe480d5ae0d97"}
Sep 17 20:58:19.711877 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.711855    2577 generic.go:356] "Generic (PLEG): container finished" podID="fa91511e-02db-479b-971f-8ae2ea22c359" containerID="cbc38a5e9c4e5ec11cb35095f2e569484a4802e48fb38fda6a035fe60b232182" exitCode=0
Sep 17 20:58:19.716277 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.716231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f" event={"ID":"fa91511e-02db-479b-971f-8ae2ea22c359","Type":"ContainerDied","Data":"cbc38a5e9c4e5ec11cb35095f2e569484a4802e48fb38fda6a035fe60b232182"}
Sep 17 20:58:19.716354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:19.716294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f" event={"ID":"fa91511e-02db-479b-971f-8ae2ea22c359","Type":"ContainerStarted","Data":"b3789a89015ff0ee1685fd8a2744f34f7f3eb7f28778880a1915efe200ac218b"}
Sep 17 20:58:20.717591 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:20.717560    2577 generic.go:356] "Generic (PLEG): container finished" podID="fa91511e-02db-479b-971f-8ae2ea22c359" containerID="e0a45b96f11c24cd22ad89428675bc7b568608e9e28f99706e1f34ffdc9b5594" exitCode=0
Sep 17 20:58:20.718018 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:20.717642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f" event={"ID":"fa91511e-02db-479b-971f-8ae2ea22c359","Type":"ContainerDied","Data":"e0a45b96f11c24cd22ad89428675bc7b568608e9e28f99706e1f34ffdc9b5594"}
Sep 17 20:58:20.719512 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:20.719487    2577 generic.go:356] "Generic (PLEG): container finished" podID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerID="7e4a82cd4372733e16ce9739d4adb88c3b8f011c3cdb0c759c2fc627626f83ce" exitCode=0
Sep 17 20:58:20.719603 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:20.719536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn" event={"ID":"c22e18cc-b0c8-4c08-904c-3a8717ee4795","Type":"ContainerDied","Data":"7e4a82cd4372733e16ce9739d4adb88c3b8f011c3cdb0c759c2fc627626f83ce"}
Sep 17 20:58:21.728022 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:21.727992    2577 generic.go:356] "Generic (PLEG): container finished" podID="fa91511e-02db-479b-971f-8ae2ea22c359" containerID="fe59e0059e1aaf7885ec453e90b3bbb4d11db9301c026421f4cf62749ca612c4" exitCode=0
Sep 17 20:58:21.728408 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:21.728074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f" event={"ID":"fa91511e-02db-479b-971f-8ae2ea22c359","Type":"ContainerDied","Data":"fe59e0059e1aaf7885ec453e90b3bbb4d11db9301c026421f4cf62749ca612c4"}
Sep 17 20:58:21.729819 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:21.729800    2577 generic.go:356] "Generic (PLEG): container finished" podID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerID="84c9ea12d56c6369320d5ef3169b7cb59ef6f8afdf4aefd11577f8a1a0fde404" exitCode=0
Sep 17 20:58:21.729918 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:21.729856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht" event={"ID":"7586f93e-91c4-4cd3-8606-4553b4b0d362","Type":"ContainerDied","Data":"84c9ea12d56c6369320d5ef3169b7cb59ef6f8afdf4aefd11577f8a1a0fde404"}
Sep 17 20:58:21.732136 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:21.732117    2577 generic.go:356] "Generic (PLEG): container finished" podID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerID="506f409ca057f612dda33eece613eabee882fc5c83a307c2de8fe7e482af5c6d" exitCode=0
Sep 17 20:58:21.732230 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:21.732190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn" event={"ID":"c22e18cc-b0c8-4c08-904c-3a8717ee4795","Type":"ContainerDied","Data":"506f409ca057f612dda33eece613eabee882fc5c83a307c2de8fe7e482af5c6d"}
Sep 17 20:58:22.737636 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.737603    2577 generic.go:356] "Generic (PLEG): container finished" podID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerID="1b4ecb5fab481f96a8a7e746f0d6831aa2773daef1f18932b2fd01d0d33f5955" exitCode=0
Sep 17 20:58:22.738087 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.737692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht" event={"ID":"7586f93e-91c4-4cd3-8606-4553b4b0d362","Type":"ContainerDied","Data":"1b4ecb5fab481f96a8a7e746f0d6831aa2773daef1f18932b2fd01d0d33f5955"}
Sep 17 20:58:22.874405 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.874381    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:22.880929 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.880905    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:22.972725 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.972690    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-bundle\") pod \"fa91511e-02db-479b-971f-8ae2ea22c359\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") "
Sep 17 20:58:22.972910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.972795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-bundle\") pod \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") "
Sep 17 20:58:22.972910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.972824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-util\") pod \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") "
Sep 17 20:58:22.972910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.972847    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz84q\" (UniqueName: \"kubernetes.io/projected/fa91511e-02db-479b-971f-8ae2ea22c359-kube-api-access-lz84q\") pod \"fa91511e-02db-479b-971f-8ae2ea22c359\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") "
Sep 17 20:58:22.972910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.972875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-util\") pod \"fa91511e-02db-479b-971f-8ae2ea22c359\" (UID: \"fa91511e-02db-479b-971f-8ae2ea22c359\") "
Sep 17 20:58:22.972910 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.972899    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwfjz\" (UniqueName: \"kubernetes.io/projected/c22e18cc-b0c8-4c08-904c-3a8717ee4795-kube-api-access-cwfjz\") pod \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\" (UID: \"c22e18cc-b0c8-4c08-904c-3a8717ee4795\") "
Sep 17 20:58:22.973273 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.973223    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-bundle" (OuterVolumeSpecName: "bundle") pod "fa91511e-02db-479b-971f-8ae2ea22c359" (UID: "fa91511e-02db-479b-971f-8ae2ea22c359"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:22.973487 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.973457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-bundle" (OuterVolumeSpecName: "bundle") pod "c22e18cc-b0c8-4c08-904c-3a8717ee4795" (UID: "c22e18cc-b0c8-4c08-904c-3a8717ee4795"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:22.975727 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.975700    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22e18cc-b0c8-4c08-904c-3a8717ee4795-kube-api-access-cwfjz" (OuterVolumeSpecName: "kube-api-access-cwfjz") pod "c22e18cc-b0c8-4c08-904c-3a8717ee4795" (UID: "c22e18cc-b0c8-4c08-904c-3a8717ee4795"). InnerVolumeSpecName "kube-api-access-cwfjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:58:22.975809 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.975737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa91511e-02db-479b-971f-8ae2ea22c359-kube-api-access-lz84q" (OuterVolumeSpecName: "kube-api-access-lz84q") pod "fa91511e-02db-479b-971f-8ae2ea22c359" (UID: "fa91511e-02db-479b-971f-8ae2ea22c359"). InnerVolumeSpecName "kube-api-access-lz84q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:58:22.978300 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.978262    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-util" (OuterVolumeSpecName: "util") pod "fa91511e-02db-479b-971f-8ae2ea22c359" (UID: "fa91511e-02db-479b-971f-8ae2ea22c359"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:22.978774 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:22.978757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-util" (OuterVolumeSpecName: "util") pod "c22e18cc-b0c8-4c08-904c-3a8717ee4795" (UID: "c22e18cc-b0c8-4c08-904c-3a8717ee4795"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:23.073686 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.073656    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:23.073686 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.073683    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22e18cc-b0c8-4c08-904c-3a8717ee4795-util\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:23.073855 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.073692    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz84q\" (UniqueName: \"kubernetes.io/projected/fa91511e-02db-479b-971f-8ae2ea22c359-kube-api-access-lz84q\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:23.073855 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.073702    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-util\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:23.073855 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.073710    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwfjz\" (UniqueName: \"kubernetes.io/projected/c22e18cc-b0c8-4c08-904c-3a8717ee4795-kube-api-access-cwfjz\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:23.073855 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.073718    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa91511e-02db-479b-971f-8ae2ea22c359-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:23.743425 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.743404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f"
Sep 17 20:58:23.743789 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.743419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7clk8f" event={"ID":"fa91511e-02db-479b-971f-8ae2ea22c359","Type":"ContainerDied","Data":"b3789a89015ff0ee1685fd8a2744f34f7f3eb7f28778880a1915efe200ac218b"}
Sep 17 20:58:23.743789 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.743461    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3789a89015ff0ee1685fd8a2744f34f7f3eb7f28778880a1915efe200ac218b"
Sep 17 20:58:23.745374 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.745351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn"
Sep 17 20:58:23.745476 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.745451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccf2jn" event={"ID":"c22e18cc-b0c8-4c08-904c-3a8717ee4795","Type":"ContainerDied","Data":"9371001e53e6af9cec3acd01a2369364475e0f429539ce2ae997b5a9fd0d7b00"}
Sep 17 20:58:23.745536 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.745505    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9371001e53e6af9cec3acd01a2369364475e0f429539ce2ae997b5a9fd0d7b00"
Sep 17 20:58:23.867754 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.867733    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:23.981341 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.981257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7487\" (UniqueName: \"kubernetes.io/projected/7586f93e-91c4-4cd3-8606-4553b4b0d362-kube-api-access-m7487\") pod \"7586f93e-91c4-4cd3-8606-4553b4b0d362\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") "
Sep 17 20:58:23.981341 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.981338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-bundle\") pod \"7586f93e-91c4-4cd3-8606-4553b4b0d362\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") "
Sep 17 20:58:23.981579 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.981377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-util\") pod \"7586f93e-91c4-4cd3-8606-4553b4b0d362\" (UID: \"7586f93e-91c4-4cd3-8606-4553b4b0d362\") "
Sep 17 20:58:23.981973 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.981933    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-bundle" (OuterVolumeSpecName: "bundle") pod "7586f93e-91c4-4cd3-8606-4553b4b0d362" (UID: "7586f93e-91c4-4cd3-8606-4553b4b0d362"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:23.983622 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.983600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7586f93e-91c4-4cd3-8606-4553b4b0d362-kube-api-access-m7487" (OuterVolumeSpecName: "kube-api-access-m7487") pod "7586f93e-91c4-4cd3-8606-4553b4b0d362" (UID: "7586f93e-91c4-4cd3-8606-4553b4b0d362"). InnerVolumeSpecName "kube-api-access-m7487". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:58:23.986553 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:23.986531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-util" (OuterVolumeSpecName: "util") pod "7586f93e-91c4-4cd3-8606-4553b4b0d362" (UID: "7586f93e-91c4-4cd3-8606-4553b4b0d362"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:24.082024 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:24.081972    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7487\" (UniqueName: \"kubernetes.io/projected/7586f93e-91c4-4cd3-8606-4553b4b0d362-kube-api-access-m7487\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:24.082024 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:24.082015    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:24.082024 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:24.082029    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7586f93e-91c4-4cd3-8606-4553b4b0d362-util\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:24.752201 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:24.752174    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht"
Sep 17 20:58:24.752201 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:24.752180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axs8ht" event={"ID":"7586f93e-91c4-4cd3-8606-4553b4b0d362","Type":"ContainerDied","Data":"2699def52cfa8f98bc3c3bc71e5ec1c23d450c799ebef978a4933c1a06923858"}
Sep 17 20:58:24.752623 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:24.752219    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2699def52cfa8f98bc3c3bc71e5ec1c23d450c799ebef978a4933c1a06923858"
Sep 17 20:58:27.766533 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:27.766497    2577 generic.go:356] "Generic (PLEG): container finished" podID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerID="ec1debdece6a666cac97cb01326262b917f5cb9e4e4436830c712252c43d3e05" exitCode=0
Sep 17 20:58:27.766920 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:27.766541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695" event={"ID":"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64","Type":"ContainerDied","Data":"ec1debdece6a666cac97cb01326262b917f5cb9e4e4436830c712252c43d3e05"}
Sep 17 20:58:28.772968 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:28.772935    2577 generic.go:356] "Generic (PLEG): container finished" podID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerID="bc2179ca40a83fff013c98f9692ff8d3a639f1eaa087eaf4c08ef01512db49d9" exitCode=0
Sep 17 20:58:28.773368 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:28.773025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695" event={"ID":"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64","Type":"ContainerDied","Data":"bc2179ca40a83fff013c98f9692ff8d3a639f1eaa087eaf4c08ef01512db49d9"}
Sep 17 20:58:29.895930 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:29.895901    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:30.032161 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.032078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr9xs\" (UniqueName: \"kubernetes.io/projected/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-kube-api-access-hr9xs\") pod \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") "
Sep 17 20:58:30.032161 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.032116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-util\") pod \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") "
Sep 17 20:58:30.032374 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.032164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-bundle\") pod \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\" (UID: \"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64\") "
Sep 17 20:58:30.032642 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.032621    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-bundle" (OuterVolumeSpecName: "bundle") pod "3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" (UID: "3815c6b8-16bb-45d5-aa29-2fe0c62c7f64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:30.034401 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.034373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-kube-api-access-hr9xs" (OuterVolumeSpecName: "kube-api-access-hr9xs") pod "3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" (UID: "3815c6b8-16bb-45d5-aa29-2fe0c62c7f64"). InnerVolumeSpecName "kube-api-access-hr9xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:58:30.036903 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.036861    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-util" (OuterVolumeSpecName: "util") pod "3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" (UID: "3815c6b8-16bb-45d5-aa29-2fe0c62c7f64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:58:30.133068 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.133019    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-bundle\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:30.133068 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.133063    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr9xs\" (UniqueName: \"kubernetes.io/projected/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-kube-api-access-hr9xs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:30.133068 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.133073    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3815c6b8-16bb-45d5-aa29-2fe0c62c7f64-util\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:58:30.782298 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.782265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695" event={"ID":"3815c6b8-16bb-45d5-aa29-2fe0c62c7f64","Type":"ContainerDied","Data":"c7ab56938c0c70fb66f5f186b039b01313f257d83a264d246eafe480d5ae0d97"}
Sep 17 20:58:30.782298 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.782287    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65t695"
Sep 17 20:58:30.782507 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:30.782309    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ab56938c0c70fb66f5f186b039b01313f257d83a264d246eafe480d5ae0d97"
Sep 17 20:58:38.016933 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.016898    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wfp95"]
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017124    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017133    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017144    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017150    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017156    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017162    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017170    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa91511e-02db-479b-971f-8ae2ea22c359" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017176    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa91511e-02db-479b-971f-8ae2ea22c359" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017184    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017189    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa91511e-02db-479b-971f-8ae2ea22c359" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017201    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa91511e-02db-479b-971f-8ae2ea22c359" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017206    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017211    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017216    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017221    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerName="util"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017227    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa91511e-02db-479b-971f-8ae2ea22c359" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017232    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa91511e-02db-479b-971f-8ae2ea22c359" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017252    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017257    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017265    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017270    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerName="pull"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017275    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017280    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017317    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3815c6b8-16bb-45d5-aa29-2fe0c62c7f64" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017324    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22e18cc-b0c8-4c08-904c-3a8717ee4795" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017332    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa91511e-02db-479b-971f-8ae2ea22c359" containerName="extract"
Sep 17 20:58:38.017342 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.017338    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7586f93e-91c4-4cd3-8606-4553b4b0d362" containerName="extract"
Sep 17 20:58:38.025708 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.025689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95"
Sep 17 20:58:38.027902 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.027883    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qrlg9\""
Sep 17 20:58:38.028338 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.028313    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 20:58:38.028690 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.028675    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 20:58:38.034592 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.034569    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wfp95"]
Sep 17 20:58:38.197315 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.197281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6hb\" (UniqueName: \"kubernetes.io/projected/d308ceef-55f2-49f6-ac50-01f31e1108cc-kube-api-access-sq6hb\") pod \"authorino-operator-6fc8f864f4-wfp95\" (UID: \"d308ceef-55f2-49f6-ac50-01f31e1108cc\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95"
Sep 17 20:58:38.298695 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.298602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6hb\" (UniqueName: \"kubernetes.io/projected/d308ceef-55f2-49f6-ac50-01f31e1108cc-kube-api-access-sq6hb\") pod \"authorino-operator-6fc8f864f4-wfp95\" (UID: \"d308ceef-55f2-49f6-ac50-01f31e1108cc\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95"
Sep 17 20:58:38.312967 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.312940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6hb\" (UniqueName: \"kubernetes.io/projected/d308ceef-55f2-49f6-ac50-01f31e1108cc-kube-api-access-sq6hb\") pod \"authorino-operator-6fc8f864f4-wfp95\" (UID: \"d308ceef-55f2-49f6-ac50-01f31e1108cc\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95"
Sep 17 20:58:38.336784 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.336752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95"
Sep 17 20:58:38.612308 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.612286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wfp95"]
Sep 17 20:58:38.614579 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:58:38.614552    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd308ceef_55f2_49f6_ac50_01f31e1108cc.slice/crio-1456f510ba18a90c63977134124c7f3a4c74f7ce26d34f23966aac40f77762b1 WatchSource:0}: Error finding container 1456f510ba18a90c63977134124c7f3a4c74f7ce26d34f23966aac40f77762b1: Status 404 returned error can't find the container with id 1456f510ba18a90c63977134124c7f3a4c74f7ce26d34f23966aac40f77762b1
Sep 17 20:58:38.818113 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:38.818080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95" event={"ID":"d308ceef-55f2-49f6-ac50-01f31e1108cc","Type":"ContainerStarted","Data":"1456f510ba18a90c63977134124c7f3a4c74f7ce26d34f23966aac40f77762b1"}
Sep 17 20:58:40.228050 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.228017    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"]
Sep 17 20:58:40.231379 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.231358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"
Sep 17 20:58:40.234028 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.234008    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zzt26\""
Sep 17 20:58:40.243891 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.243865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"]
Sep 17 20:58:40.415839 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.415808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnvq\" (UniqueName: \"kubernetes.io/projected/ff910491-41c6-4b8c-afa2-f212e1b8452e-kube-api-access-rwnvq\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-294bq\" (UID: \"ff910491-41c6-4b8c-afa2-f212e1b8452e\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"
Sep 17 20:58:40.516265 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.516208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnvq\" (UniqueName: \"kubernetes.io/projected/ff910491-41c6-4b8c-afa2-f212e1b8452e-kube-api-access-rwnvq\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-294bq\" (UID: \"ff910491-41c6-4b8c-afa2-f212e1b8452e\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"
Sep 17 20:58:40.536646 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.536624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnvq\" (UniqueName: \"kubernetes.io/projected/ff910491-41c6-4b8c-afa2-f212e1b8452e-kube-api-access-rwnvq\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-294bq\" (UID: \"ff910491-41c6-4b8c-afa2-f212e1b8452e\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"
Sep 17 20:58:40.549420 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.549380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"
Sep 17 20:58:40.830294 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.830181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95" event={"ID":"d308ceef-55f2-49f6-ac50-01f31e1108cc","Type":"ContainerStarted","Data":"e9ec79657b2c295ae35e469a3872b8010801ef5d36b8062f1acb60f5b2f75329"}
Sep 17 20:58:40.830455 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.830338    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95"
Sep 17 20:58:40.834315 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.834288    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"]
Sep 17 20:58:40.837497 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:58:40.837463    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff910491_41c6_4b8c_afa2_f212e1b8452e.slice/crio-c09068aed89477a1d2176d58afef7351eb9e402dfa56a59eab2f54f816bedea8 WatchSource:0}: Error finding container c09068aed89477a1d2176d58afef7351eb9e402dfa56a59eab2f54f816bedea8: Status 404 returned error can't find the container with id c09068aed89477a1d2176d58afef7351eb9e402dfa56a59eab2f54f816bedea8
Sep 17 20:58:40.851998 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:40.851958    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95" podStartSLOduration=2.080972217 podStartE2EDuration="3.85194348s" totalImagesPullingTime="1.770971263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:40.850455529 +0000 UTC m=+559.669747491" watchObservedRunningTime="2026-09-17 20:58:40.85194348 +0000 UTC m=+559.671235442"
Sep 17 20:58:41.835296 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:41.835258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq" event={"ID":"ff910491-41c6-4b8c-afa2-f212e1b8452e","Type":"ContainerStarted","Data":"c09068aed89477a1d2176d58afef7351eb9e402dfa56a59eab2f54f816bedea8"}
Sep 17 20:58:42.840993 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:42.840954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq" event={"ID":"ff910491-41c6-4b8c-afa2-f212e1b8452e","Type":"ContainerStarted","Data":"39ee417e9841fad7f0d5fa4a61202b177ef8a00e7d563ae30a7496afeaa7f4e7"}
Sep 17 20:58:42.841369 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:42.841077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"
Sep 17 20:58:42.859856 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:42.859806    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq" podStartSLOduration=0.962092637 podStartE2EDuration="2.85979122s" totalImagesPullingTime="1.897698583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:42.858998264 +0000 UTC m=+561.678290226" watchObservedRunningTime="2026-09-17 20:58:42.85979122 +0000 UTC m=+561.679083160"
Sep 17 20:58:46.964978 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:46.964944    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"]
Sep 17 20:58:46.968104 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:46.968089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"
Sep 17 20:58:46.970474 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:46.970448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8zbb6\""
Sep 17 20:58:46.970600 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:46.970460    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 20:58:46.982584 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:46.982556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"]
Sep 17 20:58:47.065750 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:47.065713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvz9v\" (UniqueName: \"kubernetes.io/projected/0d401026-f73a-41c2-ac49-694e36219faf-kube-api-access-qvz9v\") pod \"dns-operator-controller-manager-65dc8f76f5-vv29f\" (UID: \"0d401026-f73a-41c2-ac49-694e36219faf\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"
Sep 17 20:58:47.166656 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:47.166620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvz9v\" (UniqueName: \"kubernetes.io/projected/0d401026-f73a-41c2-ac49-694e36219faf-kube-api-access-qvz9v\") pod \"dns-operator-controller-manager-65dc8f76f5-vv29f\" (UID: \"0d401026-f73a-41c2-ac49-694e36219faf\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"
Sep 17 20:58:47.193256 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:47.193212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvz9v\" (UniqueName: \"kubernetes.io/projected/0d401026-f73a-41c2-ac49-694e36219faf-kube-api-access-qvz9v\") pod \"dns-operator-controller-manager-65dc8f76f5-vv29f\" (UID: \"0d401026-f73a-41c2-ac49-694e36219faf\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"
Sep 17 20:58:47.277972 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:47.277917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"
Sep 17 20:58:47.563545 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:47.563521    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"]
Sep 17 20:58:47.565895 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:58:47.565868    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d401026_f73a_41c2_ac49_694e36219faf.slice/crio-664bcf4d1aaac734a8d61b7161673eed8325f6ffe677d13c8fa63bd3ff644f13 WatchSource:0}: Error finding container 664bcf4d1aaac734a8d61b7161673eed8325f6ffe677d13c8fa63bd3ff644f13: Status 404 returned error can't find the container with id 664bcf4d1aaac734a8d61b7161673eed8325f6ffe677d13c8fa63bd3ff644f13
Sep 17 20:58:47.862735 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:47.862654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f" event={"ID":"0d401026-f73a-41c2-ac49-694e36219faf","Type":"ContainerStarted","Data":"664bcf4d1aaac734a8d61b7161673eed8325f6ffe677d13c8fa63bd3ff644f13"}
Sep 17 20:58:50.879831 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:50.879795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f" event={"ID":"0d401026-f73a-41c2-ac49-694e36219faf","Type":"ContainerStarted","Data":"252c347b62d7d0aca3822ae22196391870e8ed5590d47d66bf5b3e339553de18"}
Sep 17 20:58:50.880289 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:50.879918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"
Sep 17 20:58:50.898354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:50.898310    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f" podStartSLOduration=2.468184738 podStartE2EDuration="4.898296039s" totalImagesPullingTime="2.430111301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:50.897151093 +0000 UTC m=+569.716443056" watchObservedRunningTime="2026-09-17 20:58:50.898296039 +0000 UTC m=+569.717588001"
Sep 17 20:58:51.837716 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:51.837687    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-wfp95"
Sep 17 20:58:53.847645 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:58:53.847613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-294bq"
Sep 17 20:59:01.885811 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:01.885782    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vv29f"
Sep 17 20:59:36.692533 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.692498    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nqkp2"]
Sep 17 20:59:36.697249 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.697217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:36.700132 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.700114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5r9s2\""
Sep 17 20:59:36.700261 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.700116    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 20:59:36.706924 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.706901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nqkp2"]
Sep 17 20:59:36.777324 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.777290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nqkp2"]
Sep 17 20:59:36.852318 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.852287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7619897-7555-4b3e-ad24-22c043538c44-config-file\") pod \"limitador-limitador-8588749977-nqkp2\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") " pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:36.852498 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.852451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6wh\" (UniqueName: \"kubernetes.io/projected/e7619897-7555-4b3e-ad24-22c043538c44-kube-api-access-nj6wh\") pod \"limitador-limitador-8588749977-nqkp2\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") " pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:36.953297 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.953182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6wh\" (UniqueName: \"kubernetes.io/projected/e7619897-7555-4b3e-ad24-22c043538c44-kube-api-access-nj6wh\") pod \"limitador-limitador-8588749977-nqkp2\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") " pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:36.953297 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.953222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7619897-7555-4b3e-ad24-22c043538c44-config-file\") pod \"limitador-limitador-8588749977-nqkp2\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") " pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:36.953900 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.953879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7619897-7555-4b3e-ad24-22c043538c44-config-file\") pod \"limitador-limitador-8588749977-nqkp2\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") " pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:36.962323 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:36.962298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6wh\" (UniqueName: \"kubernetes.io/projected/e7619897-7555-4b3e-ad24-22c043538c44-kube-api-access-nj6wh\") pod \"limitador-limitador-8588749977-nqkp2\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") " pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:37.007163 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:37.007127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:37.281434 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:37.281408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nqkp2"]
Sep 17 20:59:37.283064 ip-10-0-134-35 kubenswrapper[2577]: W0917 20:59:37.283036    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7619897_7555_4b3e_ad24_22c043538c44.slice/crio-4dbc1ab83c935d9954048629f1d61a3e6d303c6f2c0a84c71b0286d2fcbfcb4d WatchSource:0}: Error finding container 4dbc1ab83c935d9954048629f1d61a3e6d303c6f2c0a84c71b0286d2fcbfcb4d: Status 404 returned error can't find the container with id 4dbc1ab83c935d9954048629f1d61a3e6d303c6f2c0a84c71b0286d2fcbfcb4d
Sep 17 20:59:38.089167 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:38.089129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2" event={"ID":"e7619897-7555-4b3e-ad24-22c043538c44","Type":"ContainerStarted","Data":"4dbc1ab83c935d9954048629f1d61a3e6d303c6f2c0a84c71b0286d2fcbfcb4d"}
Sep 17 20:59:39.094646 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:39.094612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2" event={"ID":"e7619897-7555-4b3e-ad24-22c043538c44","Type":"ContainerStarted","Data":"98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f"}
Sep 17 20:59:39.095112 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:39.094730    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:39.112901 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:39.112848    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2" podStartSLOduration=1.9996990129999999 podStartE2EDuration="3.112832455s" totalImagesPullingTime="1.113133442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:39.110988924 +0000 UTC m=+617.930280867" watchObservedRunningTime="2026-09-17 20:59:39.112832455 +0000 UTC m=+617.932124417"
Sep 17 20:59:50.099872 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.099841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:50.277028 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.276987    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nqkp2"]
Sep 17 20:59:50.277264 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.277201    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2" podUID="e7619897-7555-4b3e-ad24-22c043538c44" containerName="limitador" containerID="cri-o://98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f" gracePeriod=30
Sep 17 20:59:50.810125 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.810099    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:50.859212 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.859140    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj6wh\" (UniqueName: \"kubernetes.io/projected/e7619897-7555-4b3e-ad24-22c043538c44-kube-api-access-nj6wh\") pod \"e7619897-7555-4b3e-ad24-22c043538c44\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") "
Sep 17 20:59:50.859354 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.859220    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7619897-7555-4b3e-ad24-22c043538c44-config-file\") pod \"e7619897-7555-4b3e-ad24-22c043538c44\" (UID: \"e7619897-7555-4b3e-ad24-22c043538c44\") "
Sep 17 20:59:50.859545 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.859525    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7619897-7555-4b3e-ad24-22c043538c44-config-file" (OuterVolumeSpecName: "config-file") pod "e7619897-7555-4b3e-ad24-22c043538c44" (UID: "e7619897-7555-4b3e-ad24-22c043538c44"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:59:50.861454 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.861435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7619897-7555-4b3e-ad24-22c043538c44-kube-api-access-nj6wh" (OuterVolumeSpecName: "kube-api-access-nj6wh") pod "e7619897-7555-4b3e-ad24-22c043538c44" (UID: "e7619897-7555-4b3e-ad24-22c043538c44"). InnerVolumeSpecName "kube-api-access-nj6wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:59:50.960495 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.960462    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj6wh\" (UniqueName: \"kubernetes.io/projected/e7619897-7555-4b3e-ad24-22c043538c44-kube-api-access-nj6wh\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:59:50.960495 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:50.960489    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7619897-7555-4b3e-ad24-22c043538c44-config-file\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 20:59:51.148949 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.148854    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7619897-7555-4b3e-ad24-22c043538c44" containerID="98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f" exitCode=0
Sep 17 20:59:51.148949 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.148903    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2"
Sep 17 20:59:51.148949 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.148908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2" event={"ID":"e7619897-7555-4b3e-ad24-22c043538c44","Type":"ContainerDied","Data":"98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f"}
Sep 17 20:59:51.149483 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.148964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nqkp2" event={"ID":"e7619897-7555-4b3e-ad24-22c043538c44","Type":"ContainerDied","Data":"4dbc1ab83c935d9954048629f1d61a3e6d303c6f2c0a84c71b0286d2fcbfcb4d"}
Sep 17 20:59:51.149483 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.148993    2577 scope.go:117] "RemoveContainer" containerID="98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f"
Sep 17 20:59:51.157599 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.157582    2577 scope.go:117] "RemoveContainer" containerID="98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f"
Sep 17 20:59:51.157865 ip-10-0-134-35 kubenswrapper[2577]: E0917 20:59:51.157848    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f\": container with ID starting with 98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f not found: ID does not exist" containerID="98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f"
Sep 17 20:59:51.157908 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.157874    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f"} err="failed to get container status \"98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f\": rpc error: code = NotFound desc = could not find container \"98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f\": container with ID starting with 98077e9fbb4e60b73528323344c866b32afe75eb229c3fe14a0c591a6b1ee52f not found: ID does not exist"
Sep 17 20:59:51.169375 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.169352    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nqkp2"]
Sep 17 20:59:51.170999 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.170975    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nqkp2"]
Sep 17 20:59:51.716680 ip-10-0-134-35 kubenswrapper[2577]: I0917 20:59:51.716651    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7619897-7555-4b3e-ad24-22c043538c44" path="/var/lib/kubelet/pods/e7619897-7555-4b3e-ad24-22c043538c44/volumes"
Sep 17 21:00:18.551668 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.551633    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"]
Sep 17 21:00:18.552133 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.551885    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7619897-7555-4b3e-ad24-22c043538c44" containerName="limitador"
Sep 17 21:00:18.552133 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.551896    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7619897-7555-4b3e-ad24-22c043538c44" containerName="limitador"
Sep 17 21:00:18.552133 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.551951    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7619897-7555-4b3e-ad24-22c043538c44" containerName="limitador"
Sep 17 21:00:18.554715 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.554699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.577032 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.576998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"]
Sep 17 21:00:18.666896 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.666848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.666896 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.666893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.667136 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.666920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.667136 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.666940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.667136 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.667016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.667136 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.667055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.667136 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.667082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4ml\" (UniqueName: \"kubernetes.io/projected/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-kube-api-access-zf4ml\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.767630 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.767580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.767630 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.767630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.767842 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.767657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.767842 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.767695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.767842 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.767720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.767842 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.767750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.768034 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.768009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4ml\" (UniqueName: \"kubernetes.io/projected/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-kube-api-access-zf4ml\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.768522 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.768490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.770307 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.770276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.770401 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.770326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.770560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.770544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.770698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.770678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.775733 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.775707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4ml\" (UniqueName: \"kubernetes.io/projected/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-kube-api-access-zf4ml\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.775951 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.775934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4c5349-cd3f-4bbe-829e-a9110d7f82e5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vhdj6\" (UID: \"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:18.863651 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:18.863569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:19.138685 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:19.138661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"]
Sep 17 21:00:19.140752 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:00:19.140725    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4c5349_cd3f_4bbe_829e_a9110d7f82e5.slice/crio-30ddf2ac181ba92a6368106fd4d87cbce7ff24aabf35f48c9c0535682e704237 WatchSource:0}: Error finding container 30ddf2ac181ba92a6368106fd4d87cbce7ff24aabf35f48c9c0535682e704237: Status 404 returned error can't find the container with id 30ddf2ac181ba92a6368106fd4d87cbce7ff24aabf35f48c9c0535682e704237
Sep 17 21:00:19.142878 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:19.142847    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:00:19.142945 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:19.142908    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:00:19.276033 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:19.275993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6" event={"ID":"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5","Type":"ContainerStarted","Data":"bea4eaa212819be289d809a5b1969b9c6dff71a800b008449ffce867cef082ad"}
Sep 17 21:00:19.276183 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:19.276045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6" event={"ID":"5c4c5349-cd3f-4bbe-829e-a9110d7f82e5","Type":"ContainerStarted","Data":"30ddf2ac181ba92a6368106fd4d87cbce7ff24aabf35f48c9c0535682e704237"}
Sep 17 21:00:19.276183 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:19.276073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:19.297659 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:19.297594    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6" podStartSLOduration=1.297572569 podStartE2EDuration="1.297572569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:19.29599079 +0000 UTC m=+658.115282752" watchObservedRunningTime="2026-09-17 21:00:19.297572569 +0000 UTC m=+658.116864532"
Sep 17 21:00:20.283439 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.283406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vhdj6"
Sep 17 21:00:20.339428 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.339399    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"]
Sep 17 21:00:20.339609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.339589    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn" podUID="9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" containerName="discovery" containerID="cri-o://aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7" gracePeriod=30
Sep 17 21:00:20.577535 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.577510    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 21:00:20.682332 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-dns-cert\") pod \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") "
Sep 17 21:00:20.682490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-token\") pod \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") "
Sep 17 21:00:20.682490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-local-certs\") pod \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") "
Sep 17 21:00:20.682490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682394    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-kubeconfig\") pod \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") "
Sep 17 21:00:20.682490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682408    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sspvd\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-kube-api-access-sspvd\") pod \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") "
Sep 17 21:00:20.682490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-ca-configmap\") pod \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") "
Sep 17 21:00:20.682490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682457    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-cacerts\") pod \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\" (UID: \"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0\") "
Sep 17 21:00:20.683012 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.682965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" (UID: "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:00:20.684894 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.684868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" (UID: "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:00:20.685003 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.684888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-local-certs" (OuterVolumeSpecName: "local-certs") pod "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" (UID: "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:00:20.685092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.685076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" (UID: "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:20.685148 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.685114    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" (UID: "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:00:20.685290 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.685271    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-cacerts" (OuterVolumeSpecName: "cacerts") pod "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" (UID: "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:00:20.685382 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.685300    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-kube-api-access-sspvd" (OuterVolumeSpecName: "kube-api-access-sspvd") pod "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" (UID: "9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0"). InnerVolumeSpecName "kube-api-access-sspvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:20.783884 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.783852    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-token\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.783884 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.783878    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-local-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.783884 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.783886    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-kubeconfig\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.784076 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.783894    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sspvd\" (UniqueName: \"kubernetes.io/projected/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-kube-api-access-sspvd\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.784076 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.783904    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-ca-configmap\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.784076 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.783912    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-cacerts\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.784076 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:20.783920    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0-istio-csr-dns-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:00:21.286713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.286679    2577 generic.go:356] "Generic (PLEG): container finished" podID="9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" containerID="aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7" exitCode=0
Sep 17 21:00:21.287168 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.286722    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"
Sep 17 21:00:21.287168 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.286768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn" event={"ID":"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0","Type":"ContainerDied","Data":"aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7"}
Sep 17 21:00:21.287168 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.286827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn" event={"ID":"9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0","Type":"ContainerDied","Data":"95f27f7b2f97a89ecb9f90c7808a5bd0166af0a3d14dfc92e62027ecca6511c6"}
Sep 17 21:00:21.287168 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.286859    2577 scope.go:117] "RemoveContainer" containerID="aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7"
Sep 17 21:00:21.296079 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.296062    2577 scope.go:117] "RemoveContainer" containerID="aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7"
Sep 17 21:00:21.296359 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:00:21.296342    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7\": container with ID starting with aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7 not found: ID does not exist" containerID="aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7"
Sep 17 21:00:21.296430 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.296372    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7"} err="failed to get container status \"aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7\": rpc error: code = NotFound desc = could not find container \"aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7\": container with ID starting with aaa9a6561720e9daa45c052687852abd5efb185aec4102cb19b61c4542a8a8d7 not found: ID does not exist"
Sep 17 21:00:21.310495 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.310472    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"]
Sep 17 21:00:21.313016 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.312996    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xn8sn"]
Sep 17 21:00:21.718354 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:21.718324    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" path="/var/lib/kubelet/pods/9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0/volumes"
Sep 17 21:00:31.183128 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.183092    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-chksr"]
Sep 17 21:00:31.183581 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.183370    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" containerName="discovery"
Sep 17 21:00:31.183581 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.183381    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" containerName="discovery"
Sep 17 21:00:31.183581 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.183431    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cf924b7-9d12-4ae4-89c2-ae7d56e7fed0" containerName="discovery"
Sep 17 21:00:31.187686 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.187668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.192584 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.192561    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 21:00:31.194132 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.193507    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-wz7zn\""
Sep 17 21:00:31.194132 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.193544    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 21:00:31.194132 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.193552    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 21:00:31.202810 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.202787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-chksr"]
Sep 17 21:00:31.221400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.221375    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"]
Sep 17 21:00:31.224528 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.224511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.226849 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.226829    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-42mkw\""
Sep 17 21:00:31.227721 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.227696    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 21:00:31.235521 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.235501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"]
Sep 17 21:00:31.259165 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.259132    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-tdrlh"]
Sep 17 21:00:31.262332 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.262316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.265158 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.265139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 21:00:31.265249 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.265219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7hgg2\""
Sep 17 21:00:31.266203 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.266187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cfc9670-c521-4f1b-8600-2590980c6373-cert\") pod \"kserve-controller-manager-66f4b8957f-chksr\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") " pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.266277 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.266212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4zzj\" (UniqueName: \"kubernetes.io/projected/8cfc9670-c521-4f1b-8600-2590980c6373-kube-api-access-p4zzj\") pod \"kserve-controller-manager-66f4b8957f-chksr\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") " pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.274097 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.274075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tdrlh"]
Sep 17 21:00:31.367118 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.367081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtrcq\" (UniqueName: \"kubernetes.io/projected/c6277daf-bc60-4af8-8bf5-f100e77cfab8-kube-api-access-dtrcq\") pod \"llmisvc-controller-manager-79ffffc8b5-8mg29\" (UID: \"c6277daf-bc60-4af8-8bf5-f100e77cfab8\") " pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.367118 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.367127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cfc9670-c521-4f1b-8600-2590980c6373-cert\") pod \"kserve-controller-manager-66f4b8957f-chksr\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") " pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.367364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.367145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftd5\" (UniqueName: \"kubernetes.io/projected/8478205f-ddfe-4517-9c02-1aaf8ae2ecf9-kube-api-access-rftd5\") pod \"seaweedfs-64568bcd49-tdrlh\" (UID: \"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9\") " pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.367364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.367172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4zzj\" (UniqueName: \"kubernetes.io/projected/8cfc9670-c521-4f1b-8600-2590980c6373-kube-api-access-p4zzj\") pod \"kserve-controller-manager-66f4b8957f-chksr\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") " pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.367364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.367195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6277daf-bc60-4af8-8bf5-f100e77cfab8-cert\") pod \"llmisvc-controller-manager-79ffffc8b5-8mg29\" (UID: \"c6277daf-bc60-4af8-8bf5-f100e77cfab8\") " pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.367364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.367215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8478205f-ddfe-4517-9c02-1aaf8ae2ecf9-data\") pod \"seaweedfs-64568bcd49-tdrlh\" (UID: \"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9\") " pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.369700 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.369673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cfc9670-c521-4f1b-8600-2590980c6373-cert\") pod \"kserve-controller-manager-66f4b8957f-chksr\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") " pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.375854 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.375832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4zzj\" (UniqueName: \"kubernetes.io/projected/8cfc9670-c521-4f1b-8600-2590980c6373-kube-api-access-p4zzj\") pod \"kserve-controller-manager-66f4b8957f-chksr\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") " pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.468361 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.468274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6277daf-bc60-4af8-8bf5-f100e77cfab8-cert\") pod \"llmisvc-controller-manager-79ffffc8b5-8mg29\" (UID: \"c6277daf-bc60-4af8-8bf5-f100e77cfab8\") " pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.468361 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.468310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8478205f-ddfe-4517-9c02-1aaf8ae2ecf9-data\") pod \"seaweedfs-64568bcd49-tdrlh\" (UID: \"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9\") " pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.468560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.468383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtrcq\" (UniqueName: \"kubernetes.io/projected/c6277daf-bc60-4af8-8bf5-f100e77cfab8-kube-api-access-dtrcq\") pod \"llmisvc-controller-manager-79ffffc8b5-8mg29\" (UID: \"c6277daf-bc60-4af8-8bf5-f100e77cfab8\") " pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.468560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.468419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftd5\" (UniqueName: \"kubernetes.io/projected/8478205f-ddfe-4517-9c02-1aaf8ae2ecf9-kube-api-access-rftd5\") pod \"seaweedfs-64568bcd49-tdrlh\" (UID: \"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9\") " pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.468791 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.468768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8478205f-ddfe-4517-9c02-1aaf8ae2ecf9-data\") pod \"seaweedfs-64568bcd49-tdrlh\" (UID: \"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9\") " pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.470902 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.470884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6277daf-bc60-4af8-8bf5-f100e77cfab8-cert\") pod \"llmisvc-controller-manager-79ffffc8b5-8mg29\" (UID: \"c6277daf-bc60-4af8-8bf5-f100e77cfab8\") " pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.478322 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.478296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftd5\" (UniqueName: \"kubernetes.io/projected/8478205f-ddfe-4517-9c02-1aaf8ae2ecf9-kube-api-access-rftd5\") pod \"seaweedfs-64568bcd49-tdrlh\" (UID: \"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9\") " pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.478760 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.478738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtrcq\" (UniqueName: \"kubernetes.io/projected/c6277daf-bc60-4af8-8bf5-f100e77cfab8-kube-api-access-dtrcq\") pod \"llmisvc-controller-manager-79ffffc8b5-8mg29\" (UID: \"c6277daf-bc60-4af8-8bf5-f100e77cfab8\") " pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.497472 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.497444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:31.536322 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.536040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:31.571106 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.571072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:31.772402 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.772379    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-chksr"]
Sep 17 21:00:31.774343 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:00:31.774316    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfc9670_c521_4f1b_8600_2590980c6373.slice/crio-f853f69a26c48f629a51f1f2d3aea9953c03d084b0f0d817d51318bfe10c042e WatchSource:0}: Error finding container f853f69a26c48f629a51f1f2d3aea9953c03d084b0f0d817d51318bfe10c042e: Status 404 returned error can't find the container with id f853f69a26c48f629a51f1f2d3aea9953c03d084b0f0d817d51318bfe10c042e
Sep 17 21:00:31.826890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.826860    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"]
Sep 17 21:00:31.828206 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:00:31.828177    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6277daf_bc60_4af8_8bf5_f100e77cfab8.slice/crio-4e5e585b9f4ee7b709b25eaf76bf316f6a59f7fe509309f7de28ce52b4ba2740 WatchSource:0}: Error finding container 4e5e585b9f4ee7b709b25eaf76bf316f6a59f7fe509309f7de28ce52b4ba2740: Status 404 returned error can't find the container with id 4e5e585b9f4ee7b709b25eaf76bf316f6a59f7fe509309f7de28ce52b4ba2740
Sep 17 21:00:31.847667 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:31.847642    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tdrlh"]
Sep 17 21:00:31.850588 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:00:31.850562    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8478205f_ddfe_4517_9c02_1aaf8ae2ecf9.slice/crio-a6e8975e76b762730f22fdad378206bfa6175dfef1d5299d6ab38314a5903b7d WatchSource:0}: Error finding container a6e8975e76b762730f22fdad378206bfa6175dfef1d5299d6ab38314a5903b7d: Status 404 returned error can't find the container with id a6e8975e76b762730f22fdad378206bfa6175dfef1d5299d6ab38314a5903b7d
Sep 17 21:00:32.338344 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:32.338308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29" event={"ID":"c6277daf-bc60-4af8-8bf5-f100e77cfab8","Type":"ContainerStarted","Data":"4e5e585b9f4ee7b709b25eaf76bf316f6a59f7fe509309f7de28ce52b4ba2740"}
Sep 17 21:00:32.340160 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:32.340123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-chksr" event={"ID":"8cfc9670-c521-4f1b-8600-2590980c6373","Type":"ContainerStarted","Data":"f853f69a26c48f629a51f1f2d3aea9953c03d084b0f0d817d51318bfe10c042e"}
Sep 17 21:00:32.341936 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:32.341909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tdrlh" event={"ID":"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9","Type":"ContainerStarted","Data":"a6e8975e76b762730f22fdad378206bfa6175dfef1d5299d6ab38314a5903b7d"}
Sep 17 21:00:37.370563 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.370525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29" event={"ID":"c6277daf-bc60-4af8-8bf5-f100e77cfab8","Type":"ContainerStarted","Data":"d1754e709b58924aceb17b95c42e6b7b83a0e8f9a875b595cf4f9f8e7e3efe66"}
Sep 17 21:00:37.371001 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.370654    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:00:37.372144 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.372122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-chksr" event={"ID":"8cfc9670-c521-4f1b-8600-2590980c6373","Type":"ContainerStarted","Data":"d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4"}
Sep 17 21:00:37.372203 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.372173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:00:37.373600 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.373582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tdrlh" event={"ID":"8478205f-ddfe-4517-9c02-1aaf8ae2ecf9","Type":"ContainerStarted","Data":"ab1d7aabc1c0b18c3dce0d5550ea085dcc53b6b8ba6415306fdd16cef715b1bd"}
Sep 17 21:00:37.373660 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.373631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:00:37.389068 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.389021    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29" podStartSLOduration=1.916044952 podStartE2EDuration="6.389002203s" totalImagesPullingTime="4.472957251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:37.386800249 +0000 UTC m=+676.206092212" watchObservedRunningTime="2026-09-17 21:00:37.389002203 +0000 UTC m=+676.208294165"
Sep 17 21:00:37.402964 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.402904    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f4b8957f-chksr" podStartSLOduration=1.875825303 podStartE2EDuration="6.402886775s" totalImagesPullingTime="4.527061472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:37.401981014 +0000 UTC m=+676.221272977" watchObservedRunningTime="2026-09-17 21:00:37.402886775 +0000 UTC m=+676.222178739"
Sep 17 21:00:37.418743 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:37.418694    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-tdrlh" podStartSLOduration=1.878338015 podStartE2EDuration="6.418677373s" totalImagesPullingTime="4.540339358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:37.417008273 +0000 UTC m=+676.236300236" watchObservedRunningTime="2026-09-17 21:00:37.418677373 +0000 UTC m=+676.237969334"
Sep 17 21:00:43.380285 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:00:43.380229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-tdrlh"
Sep 17 21:01:08.379998 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:08.379965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-79ffffc8b5-8mg29"
Sep 17 21:01:08.382942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:08.382922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:01:16.239085 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.239007    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-chksr"]
Sep 17 21:01:16.239553 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.239203    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-66f4b8957f-chksr" podUID="8cfc9670-c521-4f1b-8600-2590980c6373" containerName="manager" containerID="cri-o://d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4" gracePeriod=10
Sep 17 21:01:16.261090 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.261066    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-zzbqz"]
Sep 17 21:01:16.264301 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.264280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.274694 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.274666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-zzbqz"]
Sep 17 21:01:16.300882 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.300851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e90ca681-208a-4549-ba1e-0bfa197927dd-cert\") pod \"kserve-controller-manager-66f4b8957f-zzbqz\" (UID: \"e90ca681-208a-4549-ba1e-0bfa197927dd\") " pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.301013 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.300912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjvp\" (UniqueName: \"kubernetes.io/projected/e90ca681-208a-4549-ba1e-0bfa197927dd-kube-api-access-vwjvp\") pod \"kserve-controller-manager-66f4b8957f-zzbqz\" (UID: \"e90ca681-208a-4549-ba1e-0bfa197927dd\") " pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.401855 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.401827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjvp\" (UniqueName: \"kubernetes.io/projected/e90ca681-208a-4549-ba1e-0bfa197927dd-kube-api-access-vwjvp\") pod \"kserve-controller-manager-66f4b8957f-zzbqz\" (UID: \"e90ca681-208a-4549-ba1e-0bfa197927dd\") " pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.402020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.401906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e90ca681-208a-4549-ba1e-0bfa197927dd-cert\") pod \"kserve-controller-manager-66f4b8957f-zzbqz\" (UID: \"e90ca681-208a-4549-ba1e-0bfa197927dd\") " pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.404187 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.404155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e90ca681-208a-4549-ba1e-0bfa197927dd-cert\") pod \"kserve-controller-manager-66f4b8957f-zzbqz\" (UID: \"e90ca681-208a-4549-ba1e-0bfa197927dd\") " pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.412213 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.412183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjvp\" (UniqueName: \"kubernetes.io/projected/e90ca681-208a-4549-ba1e-0bfa197927dd-kube-api-access-vwjvp\") pod \"kserve-controller-manager-66f4b8957f-zzbqz\" (UID: \"e90ca681-208a-4549-ba1e-0bfa197927dd\") " pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.484710 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.484689    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:01:16.502692 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.502631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cfc9670-c521-4f1b-8600-2590980c6373-cert\") pod \"8cfc9670-c521-4f1b-8600-2590980c6373\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") "
Sep 17 21:01:16.502801 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.502712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4zzj\" (UniqueName: \"kubernetes.io/projected/8cfc9670-c521-4f1b-8600-2590980c6373-kube-api-access-p4zzj\") pod \"8cfc9670-c521-4f1b-8600-2590980c6373\" (UID: \"8cfc9670-c521-4f1b-8600-2590980c6373\") "
Sep 17 21:01:16.505352 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.505322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfc9670-c521-4f1b-8600-2590980c6373-kube-api-access-p4zzj" (OuterVolumeSpecName: "kube-api-access-p4zzj") pod "8cfc9670-c521-4f1b-8600-2590980c6373" (UID: "8cfc9670-c521-4f1b-8600-2590980c6373"). InnerVolumeSpecName "kube-api-access-p4zzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:01:16.505352 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.505338    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfc9670-c521-4f1b-8600-2590980c6373-cert" (OuterVolumeSpecName: "cert") pod "8cfc9670-c521-4f1b-8600-2590980c6373" (UID: "8cfc9670-c521-4f1b-8600-2590980c6373"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:01:16.553958 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.553921    2577 generic.go:356] "Generic (PLEG): container finished" podID="8cfc9670-c521-4f1b-8600-2590980c6373" containerID="d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4" exitCode=0
Sep 17 21:01:16.554105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.553973    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-chksr"
Sep 17 21:01:16.554105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.554014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-chksr" event={"ID":"8cfc9670-c521-4f1b-8600-2590980c6373","Type":"ContainerDied","Data":"d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4"}
Sep 17 21:01:16.554105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.554071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-chksr" event={"ID":"8cfc9670-c521-4f1b-8600-2590980c6373","Type":"ContainerDied","Data":"f853f69a26c48f629a51f1f2d3aea9953c03d084b0f0d817d51318bfe10c042e"}
Sep 17 21:01:16.554105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.554094    2577 scope.go:117] "RemoveContainer" containerID="d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4"
Sep 17 21:01:16.565472 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.565447    2577 scope.go:117] "RemoveContainer" containerID="d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4"
Sep 17 21:01:16.566056 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:01:16.566029    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4\": container with ID starting with d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4 not found: ID does not exist" containerID="d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4"
Sep 17 21:01:16.566122 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.566068    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4"} err="failed to get container status \"d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4\": rpc error: code = NotFound desc = could not find container \"d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4\": container with ID starting with d75099fed25d82da58ede48428666bc99bfcca750f07ef1ca24697ed89f388a4 not found: ID does not exist"
Sep 17 21:01:16.582912 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.582885    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-chksr"]
Sep 17 21:01:16.589062 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.589035    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-chksr"]
Sep 17 21:01:16.603152 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.603133    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4zzj\" (UniqueName: \"kubernetes.io/projected/8cfc9670-c521-4f1b-8600-2590980c6373-kube-api-access-p4zzj\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:01:16.603231 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.603155    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cfc9670-c521-4f1b-8600-2590980c6373-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:01:16.620376 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.620352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:16.905389 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.904744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-zzbqz"]
Sep 17 21:01:16.910079 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:16.910058    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:01:17.561893 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:17.561847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz" event={"ID":"e90ca681-208a-4549-ba1e-0bfa197927dd","Type":"ContainerStarted","Data":"b49c92fdba4b7351334c00c773a8348a98d053e2f581d0e3689f2a20f0dc3895"}
Sep 17 21:01:17.561893 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:17.561900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz" event={"ID":"e90ca681-208a-4549-ba1e-0bfa197927dd","Type":"ContainerStarted","Data":"c6bd2e94d57f4cdac1ea428dcce210bed49b9532dcb0d3809aa5cd39b03c9c6b"}
Sep 17 21:01:17.562360 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:17.561982    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:17.581124 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:17.581078    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz" podStartSLOduration=1.237141866 podStartE2EDuration="1.581062707s" totalImagesPullingTime="343.920841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:17.578699435 +0000 UTC m=+716.397991400" watchObservedRunningTime="2026-09-17 21:01:17.581062707 +0000 UTC m=+716.400355002"
Sep 17 21:01:17.717261 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:17.717207    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cfc9670-c521-4f1b-8600-2590980c6373" path="/var/lib/kubelet/pods/8cfc9670-c521-4f1b-8600-2590980c6373/volumes"
Sep 17 21:01:48.572573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:48.572543    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f4b8957f-zzbqz"
Sep 17 21:01:49.835524 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.835484    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-x6zsn"]
Sep 17 21:01:49.836037 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.835915    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfc9670-c521-4f1b-8600-2590980c6373" containerName="manager"
Sep 17 21:01:49.836037 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.835934    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfc9670-c521-4f1b-8600-2590980c6373" containerName="manager"
Sep 17 21:01:49.836037 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.836005    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cfc9670-c521-4f1b-8600-2590980c6373" containerName="manager"
Sep 17 21:01:49.839086 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.839065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:49.841376 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.841357    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 21:01:49.841509 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.841413    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vkfvn\""
Sep 17 21:01:49.847620 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.847597    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-x6zsn"]
Sep 17 21:01:49.851076 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.851052    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-4qdtq"]
Sep 17 21:01:49.854679 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.854661    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:49.856938 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.856915    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 21:01:49.857047 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.857003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-r7pdh\""
Sep 17 21:01:49.857118 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.857082    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 21:01:49.864790 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.864769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4qdtq"]
Sep 17 21:01:49.930191 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.930156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc6197c-695e-445d-926a-bf47af035bcc-tls-certs\") pod \"model-serving-api-7b954ffc95-x6zsn\" (UID: \"9cc6197c-695e-445d-926a-bf47af035bcc\") " pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:49.930364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.930197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmz5\" (UniqueName: \"kubernetes.io/projected/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-kube-api-access-qnmz5\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:49.930364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.930266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/9cc6197c-695e-445d-926a-bf47af035bcc-kube-api-access-ws6g8\") pod \"model-serving-api-7b954ffc95-x6zsn\" (UID: \"9cc6197c-695e-445d-926a-bf47af035bcc\") " pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:49.930364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.930317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-cert\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:49.930364 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:49.930337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-metrics-certs\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.030774 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.030737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc6197c-695e-445d-926a-bf47af035bcc-tls-certs\") pod \"model-serving-api-7b954ffc95-x6zsn\" (UID: \"9cc6197c-695e-445d-926a-bf47af035bcc\") " pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:50.030774 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.030774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmz5\" (UniqueName: \"kubernetes.io/projected/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-kube-api-access-qnmz5\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.031005 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.030802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/9cc6197c-695e-445d-926a-bf47af035bcc-kube-api-access-ws6g8\") pod \"model-serving-api-7b954ffc95-x6zsn\" (UID: \"9cc6197c-695e-445d-926a-bf47af035bcc\") " pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:50.031005 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.030826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-cert\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.031005 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.030849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-metrics-certs\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.031005 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:01:50.030962    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 21:01:50.031005 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:01:50.030980    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 21:01:50.031229 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:01:50.031028    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-metrics-certs podName:03d5b668-1f6c-4b01-aa8a-1c4b889abd97 nodeName:}" failed. No retries permitted until 2026-09-17 21:01:50.531010224 +0000 UTC m=+749.350302166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-metrics-certs") pod "odh-model-controller-7b966d7984-4qdtq" (UID: "03d5b668-1f6c-4b01-aa8a-1c4b889abd97") : secret "odh-model-controller-metrics-tls" not found
Sep 17 21:01:50.031229 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:01:50.031052    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-cert podName:03d5b668-1f6c-4b01-aa8a-1c4b889abd97 nodeName:}" failed. No retries permitted until 2026-09-17 21:01:50.531034598 +0000 UTC m=+749.350326555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-cert") pod "odh-model-controller-7b966d7984-4qdtq" (UID: "03d5b668-1f6c-4b01-aa8a-1c4b889abd97") : secret "odh-model-controller-webhook-cert" not found
Sep 17 21:01:50.033409 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.033388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc6197c-695e-445d-926a-bf47af035bcc-tls-certs\") pod \"model-serving-api-7b954ffc95-x6zsn\" (UID: \"9cc6197c-695e-445d-926a-bf47af035bcc\") " pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:50.039786 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.039763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmz5\" (UniqueName: \"kubernetes.io/projected/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-kube-api-access-qnmz5\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.039879 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.039834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/9cc6197c-695e-445d-926a-bf47af035bcc-kube-api-access-ws6g8\") pod \"model-serving-api-7b954ffc95-x6zsn\" (UID: \"9cc6197c-695e-445d-926a-bf47af035bcc\") " pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:50.149735 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.149648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:50.423710 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.423671    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-x6zsn"]
Sep 17 21:01:50.426220 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:01:50.426197    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc6197c_695e_445d_926a_bf47af035bcc.slice/crio-aecbde7b2835b11d3465c68c25584c7fbcc5d5c315f6fe5bbe684db6ddf546b1 WatchSource:0}: Error finding container aecbde7b2835b11d3465c68c25584c7fbcc5d5c315f6fe5bbe684db6ddf546b1: Status 404 returned error can't find the container with id aecbde7b2835b11d3465c68c25584c7fbcc5d5c315f6fe5bbe684db6ddf546b1
Sep 17 21:01:50.535010 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.534972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-cert\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.535010 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.535009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-metrics-certs\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.537569 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.537536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-cert\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.537569 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.537548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b668-1f6c-4b01-aa8a-1c4b889abd97-metrics-certs\") pod \"odh-model-controller-7b966d7984-4qdtq\" (UID: \"03d5b668-1f6c-4b01-aa8a-1c4b889abd97\") " pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:50.714518 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.714426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-x6zsn" event={"ID":"9cc6197c-695e-445d-926a-bf47af035bcc","Type":"ContainerStarted","Data":"aecbde7b2835b11d3465c68c25584c7fbcc5d5c315f6fe5bbe684db6ddf546b1"}
Sep 17 21:01:50.767039 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:50.766999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:51.045836 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:51.045754    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4qdtq"]
Sep 17 21:01:51.048858 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:01:51.048825    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d5b668_1f6c_4b01_aa8a_1c4b889abd97.slice/crio-a80fdd24c377810095e497798651b753b1fdaf2c1e564b39acbc4b58a4ff1afd WatchSource:0}: Error finding container a80fdd24c377810095e497798651b753b1fdaf2c1e564b39acbc4b58a4ff1afd: Status 404 returned error can't find the container with id a80fdd24c377810095e497798651b753b1fdaf2c1e564b39acbc4b58a4ff1afd
Sep 17 21:01:51.722997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:51.722960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4qdtq" event={"ID":"03d5b668-1f6c-4b01-aa8a-1c4b889abd97","Type":"ContainerStarted","Data":"a80fdd24c377810095e497798651b753b1fdaf2c1e564b39acbc4b58a4ff1afd"}
Sep 17 21:01:54.741819 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:54.741782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4qdtq" event={"ID":"03d5b668-1f6c-4b01-aa8a-1c4b889abd97","Type":"ContainerStarted","Data":"702cce631aad897ccc8d73c29d77a4556c9248582de5aaccf7e471c18549ebf4"}
Sep 17 21:01:54.742315 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:54.741844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:01:54.743559 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:54.743528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-x6zsn" event={"ID":"9cc6197c-695e-445d-926a-bf47af035bcc","Type":"ContainerStarted","Data":"f109164f3cd82a46ad228439bca0d87c46543279915dd07e99af7bc65f545b25"}
Sep 17 21:01:54.743724 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:54.743705    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:01:54.758990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:54.758927    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-4qdtq" podStartSLOduration=2.761587063 podStartE2EDuration="5.758913305s" totalImagesPullingTime="2.997326242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:54.75833683 +0000 UTC m=+753.577628793" watchObservedRunningTime="2026-09-17 21:01:54.758913305 +0000 UTC m=+753.578205291"
Sep 17 21:01:54.789270 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:01:54.789208    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-x6zsn" podStartSLOduration=2.21518071 podStartE2EDuration="5.789188091s" totalImagesPullingTime="3.574007381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:54.789099536 +0000 UTC m=+753.608391499" watchObservedRunningTime="2026-09-17 21:01:54.789188091 +0000 UTC m=+753.608480055"
Sep 17 21:02:05.749535 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:02:05.749503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-4qdtq"
Sep 17 21:02:05.751598 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:02:05.751576    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-x6zsn"
Sep 17 21:03:50.839865 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.839828    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"]
Sep 17 21:03:50.843565 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.843545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:50.847060 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.847039    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-4lmq4\""
Sep 17 21:03:50.847167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.847124    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-2jd47\""
Sep 17 21:03:50.847233 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.847042    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 21:03:50.847233 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.847039    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 21:03:50.847233 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.847049    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 21:03:50.853585 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.853565    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"]
Sep 17 21:03:50.995357 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.995326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5ld\" (UniqueName: \"kubernetes.io/projected/75c22609-c773-41cd-8ee4-55463051fc24-kube-api-access-gx5ld\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:50.995523 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:50.995385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c22609-c773-41cd-8ee4-55463051fc24-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:51.096789 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:51.096704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5ld\" (UniqueName: \"kubernetes.io/projected/75c22609-c773-41cd-8ee4-55463051fc24-kube-api-access-gx5ld\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:51.096789 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:51.096766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c22609-c773-41cd-8ee4-55463051fc24-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:51.099310 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:51.099282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c22609-c773-41cd-8ee4-55463051fc24-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:51.106036 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:51.106013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5ld\" (UniqueName: \"kubernetes.io/projected/75c22609-c773-41cd-8ee4-55463051fc24-kube-api-access-gx5ld\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:51.155577 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:51.155545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:51.432559 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:51.432510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"]
Sep 17 21:03:51.435092 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:03:51.435066    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c22609_c773_41cd_8ee4_55463051fc24.slice/crio-80291a41e1b64105e3390cf8df1c3e0be60946d5762a41a680cfdacc5e9c411a WatchSource:0}: Error finding container 80291a41e1b64105e3390cf8df1c3e0be60946d5762a41a680cfdacc5e9c411a: Status 404 returned error can't find the container with id 80291a41e1b64105e3390cf8df1c3e0be60946d5762a41a680cfdacc5e9c411a
Sep 17 21:03:52.289297 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:52.289260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" event={"ID":"75c22609-c773-41cd-8ee4-55463051fc24","Type":"ContainerStarted","Data":"80291a41e1b64105e3390cf8df1c3e0be60946d5762a41a680cfdacc5e9c411a"}
Sep 17 21:03:54.300203 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:54.300174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" event={"ID":"75c22609-c773-41cd-8ee4-55463051fc24","Type":"ContainerStarted","Data":"deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0"}
Sep 17 21:03:54.300592 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:54.300290    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:03:54.322875 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:03:54.322809    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" podStartSLOduration=1.820918461 podStartE2EDuration="4.322789945s" totalImagesPullingTime="2.501871484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:03:54.319515603 +0000 UTC m=+873.138807566" watchObservedRunningTime="2026-09-17 21:03:54.322789945 +0000 UTC m=+873.142081910"
Sep 17 21:04:02.857449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.857411    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"]
Sep 17 21:04:02.860767 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.860748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:02.864464 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.864440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-ndgtw\""
Sep 17 21:04:02.864595 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.864491    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 21:04:02.864719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.864701    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 21:04:02.864719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.864707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 21:04:02.865409 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.865384    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-mfs66\""
Sep 17 21:04:02.875057 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.875034    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"]
Sep 17 21:04:02.893431 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.893368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c28ef-304d-469e-943e-4d8e966df7af-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:02.893431 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.893406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/3c3c28ef-304d-469e-943e-4d8e966df7af-kube-api-access-6br2k\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:02.994264 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.994198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c28ef-304d-469e-943e-4d8e966df7af-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:02.994264 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.994267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/3c3c28ef-304d-469e-943e-4d8e966df7af-kube-api-access-6br2k\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:02.996865 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:02.996840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c28ef-304d-469e-943e-4d8e966df7af-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:03.003945 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:03.003922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/3c3c28ef-304d-469e-943e-4d8e966df7af-kube-api-access-6br2k\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:03.173722 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:03.173639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:03.455363 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:03.455334    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"]
Sep 17 21:04:03.456930 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:04:03.456882    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3c28ef_304d_469e_943e_4d8e966df7af.slice/crio-c0a25d5a7ddaf31410b044fabd404e8f2bf2d97ed3fc293f2963b1667465cb98 WatchSource:0}: Error finding container c0a25d5a7ddaf31410b044fabd404e8f2bf2d97ed3fc293f2963b1667465cb98: Status 404 returned error can't find the container with id c0a25d5a7ddaf31410b044fabd404e8f2bf2d97ed3fc293f2963b1667465cb98
Sep 17 21:04:04.347316 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:04.347272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" event={"ID":"3c3c28ef-304d-469e-943e-4d8e966df7af","Type":"ContainerStarted","Data":"04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67"}
Sep 17 21:04:04.347734 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:04.347333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" event={"ID":"3c3c28ef-304d-469e-943e-4d8e966df7af","Type":"ContainerStarted","Data":"c0a25d5a7ddaf31410b044fabd404e8f2bf2d97ed3fc293f2963b1667465cb98"}
Sep 17 21:04:04.347734 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:04.347388    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:04.365231 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:04.365183    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" podStartSLOduration=2.365165415 podStartE2EDuration="2.365165415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:04.364008953 +0000 UTC m=+883.183300916" watchObservedRunningTime="2026-09-17 21:04:04.365165415 +0000 UTC m=+883.184457367"
Sep 17 21:04:25.306427 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:25.306345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:04:35.354124 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:35.354094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:36.319976 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:36.319940    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"]
Sep 17 21:04:38.410778 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:38.410747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"]
Sep 17 21:04:45.306432 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:45.306393    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:04:45.354355 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:45.354312    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" podUID="3c3c28ef-304d-469e-943e-4d8e966df7af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:04:49.751703 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.751666    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 21:04:49.752025 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:04:49.751703    2577 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" podUID="3c3c28ef-304d-469e-943e-4d8e966df7af" containerName="main" containerID="cri-o://04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67"
Sep 17 21:04:49.752025 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.751756    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" podUID="3c3c28ef-304d-469e-943e-4d8e966df7af" containerName="main" containerID="cri-o://04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67" gracePeriod=49
Sep 17 21:04:49.881750 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.881726    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9_3c3c28ef-304d-469e-943e-4d8e966df7af/main/0.log"
Sep 17 21:04:49.884058 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.884041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:49.949615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.949583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/3c3c28ef-304d-469e-943e-4d8e966df7af-kube-api-access-6br2k\") pod \"3c3c28ef-304d-469e-943e-4d8e966df7af\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") "
Sep 17 21:04:49.949615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.949625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c28ef-304d-469e-943e-4d8e966df7af-tls-certs\") pod \"3c3c28ef-304d-469e-943e-4d8e966df7af\" (UID: \"3c3c28ef-304d-469e-943e-4d8e966df7af\") "
Sep 17 21:04:49.951880 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.951850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3c28ef-304d-469e-943e-4d8e966df7af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c3c28ef-304d-469e-943e-4d8e966df7af" (UID: "3c3c28ef-304d-469e-943e-4d8e966df7af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:04:49.951981 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:49.951880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3c28ef-304d-469e-943e-4d8e966df7af-kube-api-access-6br2k" (OuterVolumeSpecName: "kube-api-access-6br2k") pod "3c3c28ef-304d-469e-943e-4d8e966df7af" (UID: "3c3c28ef-304d-469e-943e-4d8e966df7af"). InnerVolumeSpecName "kube-api-access-6br2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:04:50.050771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.050735    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/3c3c28ef-304d-469e-943e-4d8e966df7af-kube-api-access-6br2k\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:04:50.050771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.050770    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c28ef-304d-469e-943e-4d8e966df7af-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:04:50.568825 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.568797    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9_3c3c28ef-304d-469e-943e-4d8e966df7af/main/0.log"
Sep 17 21:04:50.569023 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.568846    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c3c28ef-304d-469e-943e-4d8e966df7af" containerID="04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67" exitCode=1
Sep 17 21:04:50.569023 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.568891    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"
Sep 17 21:04:50.569023 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.568924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" event={"ID":"3c3c28ef-304d-469e-943e-4d8e966df7af","Type":"ContainerDied","Data":"04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67"}
Sep 17 21:04:50.569023 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.568981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9" event={"ID":"3c3c28ef-304d-469e-943e-4d8e966df7af","Type":"ContainerDied","Data":"c0a25d5a7ddaf31410b044fabd404e8f2bf2d97ed3fc293f2963b1667465cb98"}
Sep 17 21:04:50.569023 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.569008    2577 scope.go:117] "RemoveContainer" containerID="04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67"
Sep 17 21:04:50.578342 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.578324    2577 scope.go:117] "RemoveContainer" containerID="04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67"
Sep 17 21:04:50.578639 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:04:50.578621    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67\": container with ID starting with 04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67 not found: ID does not exist" containerID="04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67"
Sep 17 21:04:50.578693 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.578649    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67"} err="failed to get container status \"04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67\": rpc error: code = NotFound desc = could not find container \"04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67\": container with ID starting with 04c830f8d1995ca4f7f8e03bc4683815436b5902926adb7c90901a3ccd6f4c67 not found: ID does not exist"
Sep 17 21:04:50.590181 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.590155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"]
Sep 17 21:04:50.595162 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:50.595137    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bt9v9"]
Sep 17 21:04:51.339573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:51.339524    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main" containerID="cri-o://deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0" gracePeriod=45
Sep 17 21:04:51.718616 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:51.718528    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3c28ef-304d-469e-943e-4d8e966df7af" path="/var/lib/kubelet/pods/3c3c28ef-304d-469e-943e-4d8e966df7af/volumes"
Sep 17 21:04:55.306383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:04:55.306345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:05:05.306926 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:05.306887    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:05:05.307451 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:05.307034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:05:15.307014 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:15.306966    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:05:21.593414 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.593384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:05:21.708905 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.708820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c22609-c773-41cd-8ee4-55463051fc24-tls-certs\") pod \"75c22609-c773-41cd-8ee4-55463051fc24\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") "
Sep 17 21:05:21.709073 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.708932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx5ld\" (UniqueName: \"kubernetes.io/projected/75c22609-c773-41cd-8ee4-55463051fc24-kube-api-access-gx5ld\") pod \"75c22609-c773-41cd-8ee4-55463051fc24\" (UID: \"75c22609-c773-41cd-8ee4-55463051fc24\") "
Sep 17 21:05:21.711390 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.711356    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c22609-c773-41cd-8ee4-55463051fc24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75c22609-c773-41cd-8ee4-55463051fc24" (UID: "75c22609-c773-41cd-8ee4-55463051fc24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:05:21.711539 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.711389    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c22609-c773-41cd-8ee4-55463051fc24-kube-api-access-gx5ld" (OuterVolumeSpecName: "kube-api-access-gx5ld") pod "75c22609-c773-41cd-8ee4-55463051fc24" (UID: "75c22609-c773-41cd-8ee4-55463051fc24"). InnerVolumeSpecName "kube-api-access-gx5ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:05:21.719491 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.719466    2577 generic.go:356] "Generic (PLEG): container finished" podID="75c22609-c773-41cd-8ee4-55463051fc24" containerID="deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0" exitCode=0
Sep 17 21:05:21.719592 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.719507    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"
Sep 17 21:05:21.719592 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.719545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" event={"ID":"75c22609-c773-41cd-8ee4-55463051fc24","Type":"ContainerDied","Data":"deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0"}
Sep 17 21:05:21.719685 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.719592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl" event={"ID":"75c22609-c773-41cd-8ee4-55463051fc24","Type":"ContainerDied","Data":"80291a41e1b64105e3390cf8df1c3e0be60946d5762a41a680cfdacc5e9c411a"}
Sep 17 21:05:21.719685 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.719620    2577 scope.go:117] "RemoveContainer" containerID="deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0"
Sep 17 21:05:21.728773 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.728756    2577 scope.go:117] "RemoveContainer" containerID="deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0"
Sep 17 21:05:21.729055 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:05:21.729033    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0\": container with ID starting with deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0 not found: ID does not exist" containerID="deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0"
Sep 17 21:05:21.729131 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.729061    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0"} err="failed to get container status \"deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0\": rpc error: code = NotFound desc = could not find container \"deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0\": container with ID starting with deb4eddb967cabe230b2d4244450493383272a536bc3ff0d414cf912879180a0 not found: ID does not exist"
Sep 17 21:05:21.742597 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.742574    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"]
Sep 17 21:05:21.748695 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.748676    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-6jwxl"]
Sep 17 21:05:21.809986 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.809955    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5ld\" (UniqueName: \"kubernetes.io/projected/75c22609-c773-41cd-8ee4-55463051fc24-kube-api-access-gx5ld\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:05:21.809986 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:21.809981    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c22609-c773-41cd-8ee4-55463051fc24-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:05:23.717777 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:23.717745    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75c22609-c773-41cd-8ee4-55463051fc24" path="/var/lib/kubelet/pods/75c22609-c773-41cd-8ee4-55463051fc24/volumes"
Sep 17 21:05:54.699383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.699343    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"]
Sep 17 21:05:54.699835 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.699707    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main"
Sep 17 21:05:54.699835 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.699724    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main"
Sep 17 21:05:54.699835 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.699740    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3c28ef-304d-469e-943e-4d8e966df7af" containerName="main"
Sep 17 21:05:54.699835 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.699748    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3c28ef-304d-469e-943e-4d8e966df7af" containerName="main"
Sep 17 21:05:54.699835 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.699822    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3c28ef-304d-469e-943e-4d8e966df7af" containerName="main"
Sep 17 21:05:54.699835 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.699832    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="75c22609-c773-41cd-8ee4-55463051fc24" containerName="main"
Sep 17 21:05:54.703073 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.703057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:54.706651 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.706554    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-hskjk\""
Sep 17 21:05:54.706830 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.706793    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-5cr8t\""
Sep 17 21:05:54.706977 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.706944    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 21:05:54.707109 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.706999    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 21:05:54.707177 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.707052    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 21:05:54.709836 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.709808    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"]
Sep 17 21:05:54.767308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.767276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590bf432-6d42-4b5f-8440-cc318a9904e9-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:54.767486 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.767324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8224\" (UniqueName: \"kubernetes.io/projected/590bf432-6d42-4b5f-8440-cc318a9904e9-kube-api-access-h8224\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:54.868608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.868576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8224\" (UniqueName: \"kubernetes.io/projected/590bf432-6d42-4b5f-8440-cc318a9904e9-kube-api-access-h8224\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:54.868812 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.868674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590bf432-6d42-4b5f-8440-cc318a9904e9-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:54.871369 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.871333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590bf432-6d42-4b5f-8440-cc318a9904e9-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:54.876854 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:54.876833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8224\" (UniqueName: \"kubernetes.io/projected/590bf432-6d42-4b5f-8440-cc318a9904e9-kube-api-access-h8224\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:55.015650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:55.015618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:55.286845 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:55.286820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"]
Sep 17 21:05:55.288883 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:05:55.288854    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590bf432_6d42_4b5f_8440_cc318a9904e9.slice/crio-cae573198d9fb9b37a3517584028cc694746a2b0d2b6a3e7b495261c0c935d15 WatchSource:0}: Error finding container cae573198d9fb9b37a3517584028cc694746a2b0d2b6a3e7b495261c0c935d15: Status 404 returned error can't find the container with id cae573198d9fb9b37a3517584028cc694746a2b0d2b6a3e7b495261c0c935d15
Sep 17 21:05:55.879608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:55.879573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" event={"ID":"590bf432-6d42-4b5f-8440-cc318a9904e9","Type":"ContainerStarted","Data":"c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4"}
Sep 17 21:05:55.880024 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:55.879634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" event={"ID":"590bf432-6d42-4b5f-8440-cc318a9904e9","Type":"ContainerStarted","Data":"cae573198d9fb9b37a3517584028cc694746a2b0d2b6a3e7b495261c0c935d15"}
Sep 17 21:05:55.880024 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:55.879703    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:05:55.899449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:05:55.899404    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" podStartSLOduration=1.899388876 podStartE2EDuration="1.899388876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:05:55.89831068 +0000 UTC m=+994.717602644" watchObservedRunningTime="2026-09-17 21:05:55.899388876 +0000 UTC m=+994.718680837"
Sep 17 21:06:26.886376 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:26.886346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:06:32.617196 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.617162    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"]
Sep 17 21:06:32.621925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.621898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.625045 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.625008    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 21:06:32.625190 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.625054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-b584c\""
Sep 17 21:06:32.626375 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.626354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 21:06:32.626375 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.626367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 21:06:32.626564 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.626379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-blcmj\""
Sep 17 21:06:32.636807 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.636786    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"]
Sep 17 21:06:32.679041 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.679009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.679211 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.679064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b292e26-278f-45f9-b606-5e5610d1f0b0-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.679211 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.679143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcf47\" (UniqueName: \"kubernetes.io/projected/7b292e26-278f-45f9-b606-5e5610d1f0b0-kube-api-access-wcf47\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.679211 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.679180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.779996 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.779959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b292e26-278f-45f9-b606-5e5610d1f0b0-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.780171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.780007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcf47\" (UniqueName: \"kubernetes.io/projected/7b292e26-278f-45f9-b606-5e5610d1f0b0-kube-api-access-wcf47\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.780171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.780050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.780171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.780072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.780426 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.780407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.780483 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.780442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.782596 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.782572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b292e26-278f-45f9-b606-5e5610d1f0b0-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.795217 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.795195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcf47\" (UniqueName: \"kubernetes.io/projected/7b292e26-278f-45f9-b606-5e5610d1f0b0-kube-api-access-wcf47\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:32.933345 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:32.933225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:33.218638 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:33.218565    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"]
Sep 17 21:06:33.222649 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:06:33.222621    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b292e26_278f_45f9_b606_5e5610d1f0b0.slice/crio-59d551f1cb849929728899fa41f959a3b93e8bcdf9b9fb75f5afce17931986a7 WatchSource:0}: Error finding container 59d551f1cb849929728899fa41f959a3b93e8bcdf9b9fb75f5afce17931986a7: Status 404 returned error can't find the container with id 59d551f1cb849929728899fa41f959a3b93e8bcdf9b9fb75f5afce17931986a7
Sep 17 21:06:34.060198 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:34.060155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" event={"ID":"7b292e26-278f-45f9-b606-5e5610d1f0b0","Type":"ContainerStarted","Data":"9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28"}
Sep 17 21:06:34.060650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:34.060211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" event={"ID":"7b292e26-278f-45f9-b606-5e5610d1f0b0","Type":"ContainerStarted","Data":"59d551f1cb849929728899fa41f959a3b93e8bcdf9b9fb75f5afce17931986a7"}
Sep 17 21:06:34.060650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:34.060274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:06:34.078823 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:34.078761    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" podStartSLOduration=2.078742829 podStartE2EDuration="2.078742829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:06:34.077371471 +0000 UTC m=+1032.896663430" watchObservedRunningTime="2026-09-17 21:06:34.078742829 +0000 UTC m=+1032.898034793"
Sep 17 21:06:57.829784 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:06:57.829747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"]
Sep 17 21:07:05.066799 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:05.066768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:07:06.886718 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:06.886676    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:06.909215 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:06.909178    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"]
Sep 17 21:07:12.850080 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:12.850034    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main" containerID="cri-o://c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4" gracePeriod=45
Sep 17 21:07:15.067115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:15.067076    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:16.886308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:16.886266    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:21.929080 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:21.929036    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main" containerID="cri-o://9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28" gracePeriod=45
Sep 17 21:07:25.067644 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:25.067605    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:26.886085 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:26.886049    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:26.886503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:26.886148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:07:35.066905 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:35.066869    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:35.067402 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:35.067005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:07:36.886166 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:36.886127    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:43.098674 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.098645    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:07:43.153487 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.153385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590bf432-6d42-4b5f-8440-cc318a9904e9-tls-certs\") pod \"590bf432-6d42-4b5f-8440-cc318a9904e9\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") "
Sep 17 21:07:43.153487 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.153482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8224\" (UniqueName: \"kubernetes.io/projected/590bf432-6d42-4b5f-8440-cc318a9904e9-kube-api-access-h8224\") pod \"590bf432-6d42-4b5f-8440-cc318a9904e9\" (UID: \"590bf432-6d42-4b5f-8440-cc318a9904e9\") "
Sep 17 21:07:43.155815 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.155779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590bf432-6d42-4b5f-8440-cc318a9904e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "590bf432-6d42-4b5f-8440-cc318a9904e9" (UID: "590bf432-6d42-4b5f-8440-cc318a9904e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:07:43.155924 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.155904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590bf432-6d42-4b5f-8440-cc318a9904e9-kube-api-access-h8224" (OuterVolumeSpecName: "kube-api-access-h8224") pod "590bf432-6d42-4b5f-8440-cc318a9904e9" (UID: "590bf432-6d42-4b5f-8440-cc318a9904e9"). InnerVolumeSpecName "kube-api-access-h8224". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:07:43.254552 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.254507    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590bf432-6d42-4b5f-8440-cc318a9904e9-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:07:43.254552 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.254541    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8224\" (UniqueName: \"kubernetes.io/projected/590bf432-6d42-4b5f-8440-cc318a9904e9-kube-api-access-h8224\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:07:43.392911 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.392872    2577 generic.go:356] "Generic (PLEG): container finished" podID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerID="c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4" exitCode=0
Sep 17 21:07:43.393069 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.392920    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"
Sep 17 21:07:43.393069 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.392919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" event={"ID":"590bf432-6d42-4b5f-8440-cc318a9904e9","Type":"ContainerDied","Data":"c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4"}
Sep 17 21:07:43.393069 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.393049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z" event={"ID":"590bf432-6d42-4b5f-8440-cc318a9904e9","Type":"ContainerDied","Data":"cae573198d9fb9b37a3517584028cc694746a2b0d2b6a3e7b495261c0c935d15"}
Sep 17 21:07:43.393187 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.393078    2577 scope.go:117] "RemoveContainer" containerID="c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4"
Sep 17 21:07:43.402396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.402377    2577 scope.go:117] "RemoveContainer" containerID="c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4"
Sep 17 21:07:43.402701 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:07:43.402683    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4\": container with ID starting with c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4 not found: ID does not exist" containerID="c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4"
Sep 17 21:07:43.402750 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.402710    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4"} err="failed to get container status \"c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4\": rpc error: code = NotFound desc = could not find container \"c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4\": container with ID starting with c6422d64ebccef4bfbb04fcb833197ad72e80c1304383b4d04cbaa3d3db1eac4 not found: ID does not exist"
Sep 17 21:07:43.414728 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.414669    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"]
Sep 17 21:07:43.419017 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.418995    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fxn9z"]
Sep 17 21:07:43.718657 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:43.718561    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" path="/var/lib/kubelet/pods/590bf432-6d42-4b5f-8440-cc318a9904e9/volumes"
Sep 17 21:07:45.066636 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:45.066598    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:07:52.188000 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.187972    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:07:52.330684 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.330639    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-cachi2-cache\") pod \"7b292e26-278f-45f9-b606-5e5610d1f0b0\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") "
Sep 17 21:07:52.330888 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.330721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcf47\" (UniqueName: \"kubernetes.io/projected/7b292e26-278f-45f9-b606-5e5610d1f0b0-kube-api-access-wcf47\") pod \"7b292e26-278f-45f9-b606-5e5610d1f0b0\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") "
Sep 17 21:07:52.330888 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.330749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b292e26-278f-45f9-b606-5e5610d1f0b0-tls-certs\") pod \"7b292e26-278f-45f9-b606-5e5610d1f0b0\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") "
Sep 17 21:07:52.330888 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.330792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-tokenizer-cache\") pod \"7b292e26-278f-45f9-b606-5e5610d1f0b0\" (UID: \"7b292e26-278f-45f9-b606-5e5610d1f0b0\") "
Sep 17 21:07:52.331070 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.331008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "7b292e26-278f-45f9-b606-5e5610d1f0b0" (UID: "7b292e26-278f-45f9-b606-5e5610d1f0b0"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:52.331122 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.331098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "7b292e26-278f-45f9-b606-5e5610d1f0b0" (UID: "7b292e26-278f-45f9-b606-5e5610d1f0b0"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:52.333154 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.333123    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b292e26-278f-45f9-b606-5e5610d1f0b0-kube-api-access-wcf47" (OuterVolumeSpecName: "kube-api-access-wcf47") pod "7b292e26-278f-45f9-b606-5e5610d1f0b0" (UID: "7b292e26-278f-45f9-b606-5e5610d1f0b0"). InnerVolumeSpecName "kube-api-access-wcf47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:07:52.333154 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.333128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b292e26-278f-45f9-b606-5e5610d1f0b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b292e26-278f-45f9-b606-5e5610d1f0b0" (UID: "7b292e26-278f-45f9-b606-5e5610d1f0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:07:52.432311 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.432269    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-tokenizer-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:07:52.432311 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.432305    2577 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b292e26-278f-45f9-b606-5e5610d1f0b0-cachi2-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:07:52.432311 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.432315    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcf47\" (UniqueName: \"kubernetes.io/projected/7b292e26-278f-45f9-b606-5e5610d1f0b0-kube-api-access-wcf47\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:07:52.432311 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.432324    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b292e26-278f-45f9-b606-5e5610d1f0b0-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:07:52.437684 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.437654    2577 generic.go:356] "Generic (PLEG): container finished" podID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerID="9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28" exitCode=0
Sep 17 21:07:52.437803 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.437705    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"
Sep 17 21:07:52.437803 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.437744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" event={"ID":"7b292e26-278f-45f9-b606-5e5610d1f0b0","Type":"ContainerDied","Data":"9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28"}
Sep 17 21:07:52.437803 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.437799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9" event={"ID":"7b292e26-278f-45f9-b606-5e5610d1f0b0","Type":"ContainerDied","Data":"59d551f1cb849929728899fa41f959a3b93e8bcdf9b9fb75f5afce17931986a7"}
Sep 17 21:07:52.437926 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.437822    2577 scope.go:117] "RemoveContainer" containerID="9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28"
Sep 17 21:07:52.449030 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.449006    2577 scope.go:117] "RemoveContainer" containerID="9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28"
Sep 17 21:07:52.449500 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:07:52.449479    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28\": container with ID starting with 9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28 not found: ID does not exist" containerID="9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28"
Sep 17 21:07:52.449593 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.449510    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28"} err="failed to get container status \"9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28\": rpc error: code = NotFound desc = could not find container \"9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28\": container with ID starting with 9c609cf8354ced9c62fec8ce62ebfceeaa7866c5a9211eca7ea029514db96e28 not found: ID does not exist"
Sep 17 21:07:52.458067 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.458048    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"]
Sep 17 21:07:52.461135 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:52.461115    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc9ttx9"]
Sep 17 21:07:53.717404 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:07:53.717358    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" path="/var/lib/kubelet/pods/7b292e26-278f-45f9-b606-5e5610d1f0b0/volumes"
Sep 17 21:08:11.396934 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.396903    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"]
Sep 17 21:08:11.397560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.397370    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main"
Sep 17 21:08:11.397560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.397389    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main"
Sep 17 21:08:11.397560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.397429    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main"
Sep 17 21:08:11.397560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.397437    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main"
Sep 17 21:08:11.397560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.397505    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="590bf432-6d42-4b5f-8440-cc318a9904e9" containerName="main"
Sep 17 21:08:11.397560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.397520    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b292e26-278f-45f9-b606-5e5610d1f0b0" containerName="main"
Sep 17 21:08:11.401900 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.401876    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.404502 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.404475    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 21:08:11.404642 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.404527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 21:08:11.404642 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.404613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-bzsm5\""
Sep 17 21:08:11.404830 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.404814    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 17 21:08:11.413969 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.413943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"]
Sep 17 21:08:11.485019 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.484978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485200 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485200 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485200 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxj5r\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-kube-api-access-hxj5r\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485200 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485200 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485428 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485428 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.485428 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.485311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.586568 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.586568 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.586829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxj5r\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-kube-api-access-hxj5r\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.586829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.586829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.586829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.586929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587177 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.587088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587177 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.587146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587354 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.587332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587411 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.587360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.587658 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.587640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.589222 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.589200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.589362 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.589348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.599455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.599429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxj5r\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-kube-api-access-hxj5r\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.600285 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.600267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-kwvgt\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.713928 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.713837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:11.992909 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.992880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"]
Sep 17 21:08:11.995249 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:08:11.995209    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2e3f86_b6a8_4a9f_9241_bf4bd40c1af2.slice/crio-469f67a52043c8889a2bedfff52f15bcb335151148292b7e40b31d634b33ad5e WatchSource:0}: Error finding container 469f67a52043c8889a2bedfff52f15bcb335151148292b7e40b31d634b33ad5e: Status 404 returned error can't find the container with id 469f67a52043c8889a2bedfff52f15bcb335151148292b7e40b31d634b33ad5e
Sep 17 21:08:11.997288 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:11.997271    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:08:12.542665 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:12.542621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" event={"ID":"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2","Type":"ContainerStarted","Data":"469f67a52043c8889a2bedfff52f15bcb335151148292b7e40b31d634b33ad5e"}
Sep 17 21:08:14.909215 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:14.909177    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:08:14.909537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:14.909263    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:08:14.909537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:14.909297    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:08:15.560907 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:15.560870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" event={"ID":"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2","Type":"ContainerStarted","Data":"a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954"}
Sep 17 21:08:15.583230 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:15.583169    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" podStartSLOduration=1.671572413 podStartE2EDuration="4.583147487s" totalImagesPullingTime="2.911575074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:08:15.582049229 +0000 UTC m=+1134.401341192" watchObservedRunningTime="2026-09-17 21:08:15.583147487 +0000 UTC m=+1134.402439450"
Sep 17 21:08:15.716196 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:15.716151    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" podUID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 21:08:15.717968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:15.717940    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:16.719099 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:16.719064    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:17.570414 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:17.570369    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:17.571318 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:17.571301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:08:55.962003 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.961969    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"]
Sep 17 21:08:55.965488 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.965468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:55.968932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.968897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-4z9md\""
Sep 17 21:08:55.968932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.968925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-j6wnd\""
Sep 17 21:08:55.969119 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.968950    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 21:08:55.969119 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.969016    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 21:08:55.969321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.969304    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 21:08:55.975678 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:55.975654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"]
Sep 17 21:08:56.051183 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.051147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874e9b8-e480-42a1-bb87-a018b067bd70-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.051183 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.051187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2tl\" (UniqueName: \"kubernetes.io/projected/0874e9b8-e480-42a1-bb87-a018b067bd70-kube-api-access-tn2tl\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.151593 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.151556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874e9b8-e480-42a1-bb87-a018b067bd70-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.151593 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.151596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2tl\" (UniqueName: \"kubernetes.io/projected/0874e9b8-e480-42a1-bb87-a018b067bd70-kube-api-access-tn2tl\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.154362 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.154340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874e9b8-e480-42a1-bb87-a018b067bd70-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.159492 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.159470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2tl\" (UniqueName: \"kubernetes.io/projected/0874e9b8-e480-42a1-bb87-a018b067bd70-kube-api-access-tn2tl\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.277090 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.277039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.551782 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.551744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"]
Sep 17 21:08:56.553943 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:08:56.553906    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0874e9b8_e480_42a1_bb87_a018b067bd70.slice/crio-ade7e05528cf9360ed0c34d793c35e97453c4edc0aaa8877bda2b3129957463f WatchSource:0}: Error finding container ade7e05528cf9360ed0c34d793c35e97453c4edc0aaa8877bda2b3129957463f: Status 404 returned error can't find the container with id ade7e05528cf9360ed0c34d793c35e97453c4edc0aaa8877bda2b3129957463f
Sep 17 21:08:56.759620 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.759579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" event={"ID":"0874e9b8-e480-42a1-bb87-a018b067bd70","Type":"ContainerStarted","Data":"12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b"}
Sep 17 21:08:56.759822 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.759639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" event={"ID":"0874e9b8-e480-42a1-bb87-a018b067bd70","Type":"ContainerStarted","Data":"ade7e05528cf9360ed0c34d793c35e97453c4edc0aaa8877bda2b3129957463f"}
Sep 17 21:08:56.759822 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.759667    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:08:56.778961 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:08:56.778899    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" podStartSLOduration=1.778883214 podStartE2EDuration="1.778883214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:08:56.776821463 +0000 UTC m=+1175.596113425" watchObservedRunningTime="2026-09-17 21:08:56.778883214 +0000 UTC m=+1175.598175176"
Sep 17 21:09:08.997230 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:08.997193    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"]
Sep 17 21:09:09.001252 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.001158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.003624 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.003600    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-rrptv\""
Sep 17 21:09:09.004478 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.004461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 21:09:09.009844 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.009820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"]
Sep 17 21:09:09.165550 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.165512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.165550 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.165550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.165780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.165575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.165780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.165702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.165780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.165730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7385404b-d93e-4749-8192-7ef7d1ecc00c-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.165780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.165773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvf6\" (UniqueName: \"kubernetes.io/projected/7385404b-d93e-4749-8192-7ef7d1ecc00c-kube-api-access-gsvf6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267034 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.266999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267034 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7385404b-d93e-4749-8192-7ef7d1ecc00c-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267262 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvf6\" (UniqueName: \"kubernetes.io/projected/7385404b-d93e-4749-8192-7ef7d1ecc00c-kube-api-access-gsvf6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267310 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267310 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267410 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267488 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267642 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.267726 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.267683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.269607 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.269582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.269708 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.269645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7385404b-d93e-4749-8192-7ef7d1ecc00c-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.276751 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.276725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvf6\" (UniqueName: \"kubernetes.io/projected/7385404b-d93e-4749-8192-7ef7d1ecc00c-kube-api-access-gsvf6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.315534 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.315508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:09.594988 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.594956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"]
Sep 17 21:09:09.597037 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:09:09.597012    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7385404b_d93e_4749_8192_7ef7d1ecc00c.slice/crio-8148e7a68ea396f48246007a735a92cc58fc681ee65fd09ac086370a89bfbf98 WatchSource:0}: Error finding container 8148e7a68ea396f48246007a735a92cc58fc681ee65fd09ac086370a89bfbf98: Status 404 returned error can't find the container with id 8148e7a68ea396f48246007a735a92cc58fc681ee65fd09ac086370a89bfbf98
Sep 17 21:09:09.824798 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:09.824763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" event={"ID":"7385404b-d93e-4749-8192-7ef7d1ecc00c","Type":"ContainerStarted","Data":"8148e7a68ea396f48246007a735a92cc58fc681ee65fd09ac086370a89bfbf98"}
Sep 17 21:09:11.837138 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:11.837106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" event={"ID":"7385404b-d93e-4749-8192-7ef7d1ecc00c","Type":"ContainerStarted","Data":"8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568"}
Sep 17 21:09:11.854926 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:11.854865    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" podStartSLOduration=2.159227504 podStartE2EDuration="3.854843192s" totalImagesPullingTime="1.695615688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:09:11.854110105 +0000 UTC m=+1190.673402094" watchObservedRunningTime="2026-09-17 21:09:11.854843192 +0000 UTC m=+1190.674135155"
Sep 17 21:09:12.316309 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:12.316274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:17.038121 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.038084    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"]
Sep 17 21:09:17.047552 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:09:17.047507    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:09:17.047552 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:09:17.047552 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:09:17.047552 ip-10-0-134-35 kubenswrapper[2577]:  > containerID="8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568" cmd=["/bin/sleep","15"]
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.047570    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" message="" err=<
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]:  >
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:09:17.047587    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" podUID="7385404b-d93e-4749-8192-7ef7d1ecc00c" containerName="main" containerID="cri-o://8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568"
Sep 17 21:09:17.047879 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.047636    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" podUID="7385404b-d93e-4749-8192-7ef7d1ecc00c" containerName="main" containerID="cri-o://8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568" gracePeriod=60
Sep 17 21:09:17.061321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.061293    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"]
Sep 17 21:09:17.061568 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.061539    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" podUID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" containerName="istio-proxy" containerID="cri-o://a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954" gracePeriod=30
Sep 17 21:09:17.305248 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.305198    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:17.341901 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.341867    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-model-cache\") pod \"7385404b-d93e-4749-8192-7ef7d1ecc00c\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") "
Sep 17 21:09:17.342074 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.341910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-home\") pod \"7385404b-d93e-4749-8192-7ef7d1ecc00c\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") "
Sep 17 21:09:17.342074 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.341978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-tmp-dir\") pod \"7385404b-d93e-4749-8192-7ef7d1ecc00c\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") "
Sep 17 21:09:17.342074 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.342006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-dshm\") pod \"7385404b-d93e-4749-8192-7ef7d1ecc00c\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") "
Sep 17 21:09:17.342074 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.342041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7385404b-d93e-4749-8192-7ef7d1ecc00c-tls-certs\") pod \"7385404b-d93e-4749-8192-7ef7d1ecc00c\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") "
Sep 17 21:09:17.342351 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.342080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsvf6\" (UniqueName: \"kubernetes.io/projected/7385404b-d93e-4749-8192-7ef7d1ecc00c-kube-api-access-gsvf6\") pod \"7385404b-d93e-4749-8192-7ef7d1ecc00c\" (UID: \"7385404b-d93e-4749-8192-7ef7d1ecc00c\") "
Sep 17 21:09:17.342351 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.342183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7385404b-d93e-4749-8192-7ef7d1ecc00c" (UID: "7385404b-d93e-4749-8192-7ef7d1ecc00c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:17.342351 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.342308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7385404b-d93e-4749-8192-7ef7d1ecc00c" (UID: "7385404b-d93e-4749-8192-7ef7d1ecc00c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:17.342520 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.342360    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:17.342941 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.342908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-home" (OuterVolumeSpecName: "home") pod "7385404b-d93e-4749-8192-7ef7d1ecc00c" (UID: "7385404b-d93e-4749-8192-7ef7d1ecc00c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:17.345282 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.345000    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7385404b-d93e-4749-8192-7ef7d1ecc00c-kube-api-access-gsvf6" (OuterVolumeSpecName: "kube-api-access-gsvf6") pod "7385404b-d93e-4749-8192-7ef7d1ecc00c" (UID: "7385404b-d93e-4749-8192-7ef7d1ecc00c"). InnerVolumeSpecName "kube-api-access-gsvf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:09:17.345282 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.345121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7385404b-d93e-4749-8192-7ef7d1ecc00c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7385404b-d93e-4749-8192-7ef7d1ecc00c" (UID: "7385404b-d93e-4749-8192-7ef7d1ecc00c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:09:17.345282 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.345220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-dshm" (OuterVolumeSpecName: "dshm") pod "7385404b-d93e-4749-8192-7ef7d1ecc00c" (UID: "7385404b-d93e-4749-8192-7ef7d1ecc00c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:17.442927 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.442889    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:17.442927 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.442917    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7385404b-d93e-4749-8192-7ef7d1ecc00c-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:17.442927 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.442929    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvf6\" (UniqueName: \"kubernetes.io/projected/7385404b-d93e-4749-8192-7ef7d1ecc00c-kube-api-access-gsvf6\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:17.442927 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.442937    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:17.443287 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.442950    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7385404b-d93e-4749-8192-7ef7d1ecc00c-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:17.877828 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.877788    2577 generic.go:356] "Generic (PLEG): container finished" podID="7385404b-d93e-4749-8192-7ef7d1ecc00c" containerID="8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568" exitCode=0
Sep 17 21:09:17.877994 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.877853    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"
Sep 17 21:09:17.877994 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.877875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" event={"ID":"7385404b-d93e-4749-8192-7ef7d1ecc00c","Type":"ContainerDied","Data":"8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568"}
Sep 17 21:09:17.877994 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.877932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65" event={"ID":"7385404b-d93e-4749-8192-7ef7d1ecc00c","Type":"ContainerDied","Data":"8148e7a68ea396f48246007a735a92cc58fc681ee65fd09ac086370a89bfbf98"}
Sep 17 21:09:17.877994 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.877954    2577 scope.go:117] "RemoveContainer" containerID="8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568"
Sep 17 21:09:17.887321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.887302    2577 scope.go:117] "RemoveContainer" containerID="8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568"
Sep 17 21:09:17.887618 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:09:17.887596    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568\": container with ID starting with 8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568 not found: ID does not exist" containerID="8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568"
Sep 17 21:09:17.887672 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.887637    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568"} err="failed to get container status \"8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568\": rpc error: code = NotFound desc = could not find container \"8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568\": container with ID starting with 8b6ee1ba86604ed03b1c0b77865167ca6ef1db8301200a7b823615071b80c568 not found: ID does not exist"
Sep 17 21:09:17.893798 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.893775    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"]
Sep 17 21:09:17.897314 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:17.897291    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbgw65"]
Sep 17 21:09:18.612447 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:18.612403    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" podUID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 21:09:19.411087 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.411060    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:09:19.456586 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-socket\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.456586 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxj5r\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-kube-api-access-hxj5r\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.456586 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-envoy\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.456863 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-data\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.456924 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456881    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-certs\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.456924 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:19.457036 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:19.457036 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-podinfo\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.457036 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.456978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-credential-socket\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.457305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-token\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.457305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istiod-ca-cert\") pod \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\" (UID: \"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2\") "
Sep 17 21:09:19.457305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457115    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:19.457534 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457437    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.457534 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457458    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-workload-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.457534 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457462    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:19.457534 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457471    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-data\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.457907 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.457531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:19.459328 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.459279    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-kube-api-access-hxj5r" (OuterVolumeSpecName: "kube-api-access-hxj5r") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "kube-api-access-hxj5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:09:19.459617 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.459594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:19.459803 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.459781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 21:09:19.459916 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.459880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" (UID: "7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:09:19.558067 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.558038    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxj5r\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-kube-api-access-hxj5r\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.558067 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.558062    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-envoy\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.558067 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.558074    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-podinfo\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.558305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.558081    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-credential-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.558305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.558089    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istio-token\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.558305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.558097    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2-istiod-ca-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.717562 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.717483    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7385404b-d93e-4749-8192-7ef7d1ecc00c" path="/var/lib/kubelet/pods/7385404b-d93e-4749-8192-7ef7d1ecc00c/volumes"
Sep 17 21:09:19.889777 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.889743    2577 generic.go:356] "Generic (PLEG): container finished" podID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" containerID="a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954" exitCode=0
Sep 17 21:09:19.889942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.889803    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"
Sep 17 21:09:19.889942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.889832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" event={"ID":"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2","Type":"ContainerDied","Data":"a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954"}
Sep 17 21:09:19.889942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.889887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt" event={"ID":"7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2","Type":"ContainerDied","Data":"469f67a52043c8889a2bedfff52f15bcb335151148292b7e40b31d634b33ad5e"}
Sep 17 21:09:19.889942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.889916    2577 scope.go:117] "RemoveContainer" containerID="a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954"
Sep 17 21:09:19.901704 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.901682    2577 scope.go:117] "RemoveContainer" containerID="a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954"
Sep 17 21:09:19.902063 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:09:19.902035    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954\": container with ID starting with a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954 not found: ID does not exist" containerID="a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954"
Sep 17 21:09:19.902163 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.902067    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954"} err="failed to get container status \"a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954\": rpc error: code = NotFound desc = could not find container \"a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954\": container with ID starting with a90717d09c6b88a9193f35ef8c735abd734c79a05eda9535befad77af9e1a954 not found: ID does not exist"
Sep 17 21:09:19.908996 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.908971    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"]
Sep 17 21:09:19.912487 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:19.912467    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-kwvgt"]
Sep 17 21:09:21.717385 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:21.717352    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" path="/var/lib/kubelet/pods/7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2/volumes"
Sep 17 21:09:27.766549 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:27.766516    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:09:47.407518 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.407478    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"]
Sep 17 21:09:47.407949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.407817    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" containerName="istio-proxy"
Sep 17 21:09:47.407949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.407830    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" containerName="istio-proxy"
Sep 17 21:09:47.407949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.407837    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7385404b-d93e-4749-8192-7ef7d1ecc00c" containerName="main"
Sep 17 21:09:47.407949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.407842    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7385404b-d93e-4749-8192-7ef7d1ecc00c" containerName="main"
Sep 17 21:09:47.407949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.407889    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2e3f86-b6a8-4a9f-9241-bf4bd40c1af2" containerName="istio-proxy"
Sep 17 21:09:47.407949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.407898    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7385404b-d93e-4749-8192-7ef7d1ecc00c" containerName="main"
Sep 17 21:09:47.415642 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.415618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:47.419476 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.419446    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 21:09:47.419608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.419460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-226qh\""
Sep 17 21:09:47.419608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.419494    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 21:09:47.419608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.419463    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-grzv6\""
Sep 17 21:09:47.419855 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.419634    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 21:09:47.420848 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.420825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"]
Sep 17 21:09:47.486698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.486660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnc7x\" (UniqueName: \"kubernetes.io/projected/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-kube-api-access-mnc7x\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:47.486882 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.486756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:47.587151 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.587119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:47.587434 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.587167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc7x\" (UniqueName: \"kubernetes.io/projected/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-kube-api-access-mnc7x\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:47.589885 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.589860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:47.594870 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.594845    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnc7x\" (UniqueName: \"kubernetes.io/projected/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-kube-api-access-mnc7x\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:47.726962 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:47.726887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:48.003845 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:48.003820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"]
Sep 17 21:09:48.005720 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:09:48.005697    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b056e6_8c18_40e7_81e1_0b3f0d7cfa56.slice/crio-adaddc99f189867f4628aa4336e8e5a8d70eaacca3c91aea9fae2d62779174da WatchSource:0}: Error finding container adaddc99f189867f4628aa4336e8e5a8d70eaacca3c91aea9fae2d62779174da: Status 404 returned error can't find the container with id adaddc99f189867f4628aa4336e8e5a8d70eaacca3c91aea9fae2d62779174da
Sep 17 21:09:48.030030 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:48.029997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" event={"ID":"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56","Type":"ContainerStarted","Data":"adaddc99f189867f4628aa4336e8e5a8d70eaacca3c91aea9fae2d62779174da"}
Sep 17 21:09:49.036673 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:49.036633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" event={"ID":"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56","Type":"ContainerStarted","Data":"6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b"}
Sep 17 21:09:49.037083 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:49.036728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:09:49.054085 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:09:49.054033    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" podStartSLOduration=2.054016339 podStartE2EDuration="2.054016339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:09:49.052526569 +0000 UTC m=+1227.871818534" watchObservedRunningTime="2026-09-17 21:09:49.054016339 +0000 UTC m=+1227.873308339"
Sep 17 21:10:20.044653 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:10:20.044575    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:10:56.489614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:10:56.489581    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"]
Sep 17 21:10:57.766451 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:10:57.766416    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:11:07.766304 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:07.766263    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:11:11.509568 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:11.509515    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main" containerID="cri-o://12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b" gracePeriod=45
Sep 17 21:11:17.765829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:17.765785    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:11:17.766298 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:17.765908    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:11:27.766272 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:27.766214    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:11:37.766786 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:37.766743    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:11:41.758664 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:41.758633    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:11:41.825461 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:41.825367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874e9b8-e480-42a1-bb87-a018b067bd70-tls-certs\") pod \"0874e9b8-e480-42a1-bb87-a018b067bd70\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") "
Sep 17 21:11:41.825461 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:41.825427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn2tl\" (UniqueName: \"kubernetes.io/projected/0874e9b8-e480-42a1-bb87-a018b067bd70-kube-api-access-tn2tl\") pod \"0874e9b8-e480-42a1-bb87-a018b067bd70\" (UID: \"0874e9b8-e480-42a1-bb87-a018b067bd70\") "
Sep 17 21:11:41.827783 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:41.827753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0874e9b8-e480-42a1-bb87-a018b067bd70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0874e9b8-e480-42a1-bb87-a018b067bd70" (UID: "0874e9b8-e480-42a1-bb87-a018b067bd70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:11:41.827903 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:41.827785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0874e9b8-e480-42a1-bb87-a018b067bd70-kube-api-access-tn2tl" (OuterVolumeSpecName: "kube-api-access-tn2tl") pod "0874e9b8-e480-42a1-bb87-a018b067bd70" (UID: "0874e9b8-e480-42a1-bb87-a018b067bd70"). InnerVolumeSpecName "kube-api-access-tn2tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:11:41.926606 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:41.926571    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2tl\" (UniqueName: \"kubernetes.io/projected/0874e9b8-e480-42a1-bb87-a018b067bd70-kube-api-access-tn2tl\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:11:41.926606 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:41.926600    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874e9b8-e480-42a1-bb87-a018b067bd70-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:11:42.576171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.576131    2577 generic.go:356] "Generic (PLEG): container finished" podID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerID="12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b" exitCode=0
Sep 17 21:11:42.576171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.576171    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"
Sep 17 21:11:42.576402 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.576217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" event={"ID":"0874e9b8-e480-42a1-bb87-a018b067bd70","Type":"ContainerDied","Data":"12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b"}
Sep 17 21:11:42.576402 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.576281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v" event={"ID":"0874e9b8-e480-42a1-bb87-a018b067bd70","Type":"ContainerDied","Data":"ade7e05528cf9360ed0c34d793c35e97453c4edc0aaa8877bda2b3129957463f"}
Sep 17 21:11:42.576402 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.576304    2577 scope.go:117] "RemoveContainer" containerID="12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b"
Sep 17 21:11:42.585535 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.585520    2577 scope.go:117] "RemoveContainer" containerID="12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b"
Sep 17 21:11:42.585799 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:11:42.585781    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b\": container with ID starting with 12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b not found: ID does not exist" containerID="12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b"
Sep 17 21:11:42.585890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.585805    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b"} err="failed to get container status \"12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b\": rpc error: code = NotFound desc = could not find container \"12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b\": container with ID starting with 12ce7f6f3dd4deaac5d05b4d36a1d0c5f951615a9d454f857c908daec989365b not found: ID does not exist"
Sep 17 21:11:42.598271 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.598227    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"]
Sep 17 21:11:42.604179 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:42.604157    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758kn8v"]
Sep 17 21:11:43.718588 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:11:43.718504    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" path="/var/lib/kubelet/pods/0874e9b8-e480-42a1-bb87-a018b067bd70/volumes"
Sep 17 21:12:05.575342 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.575308    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"]
Sep 17 21:12:05.575894 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.575758    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main"
Sep 17 21:12:05.575894 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.575777    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main"
Sep 17 21:12:05.575894 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.575855    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0874e9b8-e480-42a1-bb87-a018b067bd70" containerName="main"
Sep 17 21:12:05.578944 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.578922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.582276 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.582250    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 21:12:05.582276 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.582231    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 21:12:05.582441 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.582231    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-l6fbg\""
Sep 17 21:12:05.582441 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.582301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 21:12:05.588544 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.588520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"]
Sep 17 21:12:05.716746 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.716717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.716925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.716753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ee448-f602-424f-8d8f-745e1e566d99-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.716925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.716789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlm4f\" (UniqueName: \"kubernetes.io/projected/9d7ee448-f602-424f-8d8f-745e1e566d99-kube-api-access-xlm4f\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.716925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.716834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.716925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.716865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.716925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.716887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.784573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.784532    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"]
Sep 17 21:12:05.787809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.787786    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:05.790014 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.789991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-7mhln\""
Sep 17 21:12:05.796998 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.796976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"]
Sep 17 21:12:05.817819 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.817789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.817958 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.817836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ee448-f602-424f-8d8f-745e1e566d99-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.818395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.818051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlm4f\" (UniqueName: \"kubernetes.io/projected/9d7ee448-f602-424f-8d8f-745e1e566d99-kube-api-access-xlm4f\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.818395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.818128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.818395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.818206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.818395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.818292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.818395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.818340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.818719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.818439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.818719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.818588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.820692 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.820669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.820917 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.820902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ee448-f602-424f-8d8f-745e1e566d99-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.825561 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.825513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlm4f\" (UniqueName: \"kubernetes.io/projected/9d7ee448-f602-424f-8d8f-745e1e566d99-kube-api-access-xlm4f\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.889600 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.889566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:05.918811 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.918776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1eedb4-36c8-4533-9558-c7435628e9bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:05.918968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:05.918826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwtw\" (UniqueName: \"kubernetes.io/projected/dc1eedb4-36c8-4533-9558-c7435628e9bc-kube-api-access-vgwtw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:06.019444 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.019403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwtw\" (UniqueName: \"kubernetes.io/projected/dc1eedb4-36c8-4533-9558-c7435628e9bc-kube-api-access-vgwtw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:06.019654 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.019538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1eedb4-36c8-4533-9558-c7435628e9bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:06.028156 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.022674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1eedb4-36c8-4533-9558-c7435628e9bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:06.031287 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.031263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwtw\" (UniqueName: \"kubernetes.io/projected/dc1eedb4-36c8-4533-9558-c7435628e9bc-kube-api-access-vgwtw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:06.098440 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.098360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:06.167447 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.167416    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"]
Sep 17 21:12:06.169087 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:12:06.169056    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7ee448_f602_424f_8d8f_745e1e566d99.slice/crio-88f548783f1f179acaccb5f9a96e5c326bd4c458af3e0a312d55222358ce3403 WatchSource:0}: Error finding container 88f548783f1f179acaccb5f9a96e5c326bd4c458af3e0a312d55222358ce3403: Status 404 returned error can't find the container with id 88f548783f1f179acaccb5f9a96e5c326bd4c458af3e0a312d55222358ce3403
Sep 17 21:12:06.371764 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.371738    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"]
Sep 17 21:12:06.373552 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:12:06.373524    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1eedb4_36c8_4533_9558_c7435628e9bc.slice/crio-ef9f29bc2460e3252052ffada25093338312afc24c34daf91c42002376b72fa7 WatchSource:0}: Error finding container ef9f29bc2460e3252052ffada25093338312afc24c34daf91c42002376b72fa7: Status 404 returned error can't find the container with id ef9f29bc2460e3252052ffada25093338312afc24c34daf91c42002376b72fa7
Sep 17 21:12:06.690359 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.690267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" event={"ID":"dc1eedb4-36c8-4533-9558-c7435628e9bc","Type":"ContainerStarted","Data":"cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba"}
Sep 17 21:12:06.690359 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.690321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" event={"ID":"dc1eedb4-36c8-4533-9558-c7435628e9bc","Type":"ContainerStarted","Data":"ef9f29bc2460e3252052ffada25093338312afc24c34daf91c42002376b72fa7"}
Sep 17 21:12:06.690783 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.690362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:06.691823 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.691799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" event={"ID":"9d7ee448-f602-424f-8d8f-745e1e566d99","Type":"ContainerStarted","Data":"4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84"}
Sep 17 21:12:06.691881 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.691847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" event={"ID":"9d7ee448-f602-424f-8d8f-745e1e566d99","Type":"ContainerStarted","Data":"88f548783f1f179acaccb5f9a96e5c326bd4c458af3e0a312d55222358ce3403"}
Sep 17 21:12:06.709641 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.709591    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" podStartSLOduration=1.709576718 podStartE2EDuration="1.709576718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:12:06.707548396 +0000 UTC m=+1365.526840359" watchObservedRunningTime="2026-09-17 21:12:06.709576718 +0000 UTC m=+1365.528868680"
Sep 17 21:12:06.731274 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.731217    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" podStartSLOduration=1.731200589 podStartE2EDuration="1.731200589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:12:06.728105579 +0000 UTC m=+1365.547397541" watchObservedRunningTime="2026-09-17 21:12:06.731200589 +0000 UTC m=+1365.550492550"
Sep 17 21:12:06.889859 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:06.889828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:15.890009 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:15.889977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:15.902901 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:15.902872    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:16.750939 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:16.750911    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:37.699257 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:37.699215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:12:50.301806 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:50.301765    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"]
Sep 17 21:12:53.250969 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.250927    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"]
Sep 17 21:12:53.256437 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.256405    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"]
Sep 17 21:12:53.262327 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:12:53.262287    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:12:53.262327 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:12:53.262327 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:12:53.262327 ip-10-0-134-35 kubenswrapper[2577]:  > containerID="4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84" cmd=["/bin/sleep","15"]
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.262346    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" message="" err=<
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]:  >
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:12:53.262360    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" podUID="9d7ee448-f602-424f-8d8f-745e1e566d99" containerName="main" containerID="cri-o://4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84"
Sep 17 21:12:53.262630 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.262422    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" podUID="9d7ee448-f602-424f-8d8f-745e1e566d99" containerName="main" containerID="cri-o://4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84" gracePeriod=60
Sep 17 21:12:53.615416 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.615390    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:53.715041 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-home\") pod \"9d7ee448-f602-424f-8d8f-745e1e566d99\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") "
Sep 17 21:12:53.715041 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlm4f\" (UniqueName: \"kubernetes.io/projected/9d7ee448-f602-424f-8d8f-745e1e566d99-kube-api-access-xlm4f\") pod \"9d7ee448-f602-424f-8d8f-745e1e566d99\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") "
Sep 17 21:12:53.715324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-dshm\") pod \"9d7ee448-f602-424f-8d8f-745e1e566d99\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") "
Sep 17 21:12:53.715324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-model-cache\") pod \"9d7ee448-f602-424f-8d8f-745e1e566d99\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") "
Sep 17 21:12:53.715324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715183    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ee448-f602-424f-8d8f-745e1e566d99-tls-certs\") pod \"9d7ee448-f602-424f-8d8f-745e1e566d99\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") "
Sep 17 21:12:53.715324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-tmp-dir\") pod \"9d7ee448-f602-424f-8d8f-745e1e566d99\" (UID: \"9d7ee448-f602-424f-8d8f-745e1e566d99\") "
Sep 17 21:12:53.715538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-home" (OuterVolumeSpecName: "home") pod "9d7ee448-f602-424f-8d8f-745e1e566d99" (UID: "9d7ee448-f602-424f-8d8f-745e1e566d99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.715538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715451    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.715538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715453    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d7ee448-f602-424f-8d8f-745e1e566d99" (UID: "9d7ee448-f602-424f-8d8f-745e1e566d99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.715538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.715494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d7ee448-f602-424f-8d8f-745e1e566d99" (UID: "9d7ee448-f602-424f-8d8f-745e1e566d99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.717569 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.717543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-dshm" (OuterVolumeSpecName: "dshm") pod "9d7ee448-f602-424f-8d8f-745e1e566d99" (UID: "9d7ee448-f602-424f-8d8f-745e1e566d99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.717694 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.717601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7ee448-f602-424f-8d8f-745e1e566d99-kube-api-access-xlm4f" (OuterVolumeSpecName: "kube-api-access-xlm4f") pod "9d7ee448-f602-424f-8d8f-745e1e566d99" (UID: "9d7ee448-f602-424f-8d8f-745e1e566d99"). InnerVolumeSpecName "kube-api-access-xlm4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:12:53.717750 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.717706    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7ee448-f602-424f-8d8f-745e1e566d99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d7ee448-f602-424f-8d8f-745e1e566d99" (UID: "9d7ee448-f602-424f-8d8f-745e1e566d99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:12:53.816890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.816815    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ee448-f602-424f-8d8f-745e1e566d99-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.816890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.816841    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.816890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.816849    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlm4f\" (UniqueName: \"kubernetes.io/projected/9d7ee448-f602-424f-8d8f-745e1e566d99-kube-api-access-xlm4f\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.816890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.816857    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.816890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.816865    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ee448-f602-424f-8d8f-745e1e566d99-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.926713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.924990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" event={"ID":"9d7ee448-f602-424f-8d8f-745e1e566d99","Type":"ContainerDied","Data":"4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84"}
Sep 17 21:12:53.926713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.925371    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d7ee448-f602-424f-8d8f-745e1e566d99" containerID="4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84" exitCode=0
Sep 17 21:12:53.926713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.925404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"
Sep 17 21:12:53.926713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.925419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5" event={"ID":"9d7ee448-f602-424f-8d8f-745e1e566d99","Type":"ContainerDied","Data":"88f548783f1f179acaccb5f9a96e5c326bd4c458af3e0a312d55222358ce3403"}
Sep 17 21:12:53.926713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.925441    2577 scope.go:117] "RemoveContainer" containerID="4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84"
Sep 17 21:12:53.939550 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.939529    2577 scope.go:117] "RemoveContainer" containerID="4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84"
Sep 17 21:12:53.939863 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:12:53.939839    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84\": container with ID starting with 4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84 not found: ID does not exist" containerID="4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84"
Sep 17 21:12:53.939927 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.939877    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84"} err="failed to get container status \"4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84\": rpc error: code = NotFound desc = could not find container \"4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84\": container with ID starting with 4a02fb40152e657fae2dc5eee66fe0e5a06ffadf0b754970204f5d1224f76e84 not found: ID does not exist"
Sep 17 21:12:53.949605 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.949580    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"]
Sep 17 21:12:53.957252 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:53.957218    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vhml5"]
Sep 17 21:12:55.717795 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:55.717760    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d7ee448-f602-424f-8d8f-745e1e566d99" path="/var/lib/kubelet/pods/9d7ee448-f602-424f-8d8f-745e1e566d99/volumes"
Sep 17 21:12:57.699637 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:12:57.699600    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:00.044681 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:00.044647    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:05.321977 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:05.321929    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main" containerID="cri-o://6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b" gracePeriod=45
Sep 17 21:13:07.699838 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:07.699797    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:08.278092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:08.278030    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main" containerID="cri-o://cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba" gracePeriod=45
Sep 17 21:13:10.044615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:10.044573    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:17.699492 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:17.699409    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:17.699946 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:17.699537    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:13:20.043931 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:20.043895    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:20.044390 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:20.044048    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:13:27.699034 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:27.698992    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:30.044622 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:30.044584    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:35.576839 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:35.576813    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:13:35.642153 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:35.642052    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-tls-certs\") pod \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") "
Sep 17 21:13:35.642427 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:35.642164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnc7x\" (UniqueName: \"kubernetes.io/projected/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-kube-api-access-mnc7x\") pod \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\" (UID: \"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56\") "
Sep 17 21:13:35.644480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:35.644451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-kube-api-access-mnc7x" (OuterVolumeSpecName: "kube-api-access-mnc7x") pod "26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" (UID: "26b056e6-8c18-40e7-81e1-0b3f0d7cfa56"). InnerVolumeSpecName "kube-api-access-mnc7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:13:35.644580 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:35.644480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" (UID: "26b056e6-8c18-40e7-81e1-0b3f0d7cfa56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:13:35.743883 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:35.743843    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:13:35.743883 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:35.743875    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnc7x\" (UniqueName: \"kubernetes.io/projected/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56-kube-api-access-mnc7x\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:13:36.126784 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.126751    2577 generic.go:356] "Generic (PLEG): container finished" podID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerID="6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b" exitCode=0
Sep 17 21:13:36.126968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.126794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" event={"ID":"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56","Type":"ContainerDied","Data":"6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b"}
Sep 17 21:13:36.126968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.126797    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"
Sep 17 21:13:36.126968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.126835    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss" event={"ID":"26b056e6-8c18-40e7-81e1-0b3f0d7cfa56","Type":"ContainerDied","Data":"adaddc99f189867f4628aa4336e8e5a8d70eaacca3c91aea9fae2d62779174da"}
Sep 17 21:13:36.126968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.126856    2577 scope.go:117] "RemoveContainer" containerID="6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b"
Sep 17 21:13:36.135459 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.135438    2577 scope.go:117] "RemoveContainer" containerID="6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b"
Sep 17 21:13:36.135770 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:13:36.135738    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b\": container with ID starting with 6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b not found: ID does not exist" containerID="6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b"
Sep 17 21:13:36.135837 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.135783    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b"} err="failed to get container status \"6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b\": rpc error: code = NotFound desc = could not find container \"6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b\": container with ID starting with 6755c942a928f720d28a73daa21d7a711e7bd998ec3940b0e6f710022a63151b not found: ID does not exist"
Sep 17 21:13:36.149441 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.149416    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"]
Sep 17 21:13:36.156657 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:36.156630    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-j7dss"]
Sep 17 21:13:37.698925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:37.698879    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:13:37.717570 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:37.717545    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" path="/var/lib/kubelet/pods/26b056e6-8c18-40e7-81e1-0b3f0d7cfa56/volumes"
Sep 17 21:13:38.529090 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:38.529064    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:13:38.568131 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:38.568046    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgwtw\" (UniqueName: \"kubernetes.io/projected/dc1eedb4-36c8-4533-9558-c7435628e9bc-kube-api-access-vgwtw\") pod \"dc1eedb4-36c8-4533-9558-c7435628e9bc\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") "
Sep 17 21:13:38.568131 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:38.568101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1eedb4-36c8-4533-9558-c7435628e9bc-tls-certs\") pod \"dc1eedb4-36c8-4533-9558-c7435628e9bc\" (UID: \"dc1eedb4-36c8-4533-9558-c7435628e9bc\") "
Sep 17 21:13:38.570301 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:38.570270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1eedb4-36c8-4533-9558-c7435628e9bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc1eedb4-36c8-4533-9558-c7435628e9bc" (UID: "dc1eedb4-36c8-4533-9558-c7435628e9bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:13:38.570400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:38.570295    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1eedb4-36c8-4533-9558-c7435628e9bc-kube-api-access-vgwtw" (OuterVolumeSpecName: "kube-api-access-vgwtw") pod "dc1eedb4-36c8-4533-9558-c7435628e9bc" (UID: "dc1eedb4-36c8-4533-9558-c7435628e9bc"). InnerVolumeSpecName "kube-api-access-vgwtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:13:38.669416 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:38.669382    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgwtw\" (UniqueName: \"kubernetes.io/projected/dc1eedb4-36c8-4533-9558-c7435628e9bc-kube-api-access-vgwtw\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:13:38.669416 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:38.669410    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1eedb4-36c8-4533-9558-c7435628e9bc-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:13:39.144450 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.144412    2577 generic.go:356] "Generic (PLEG): container finished" podID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerID="cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba" exitCode=0
Sep 17 21:13:39.144892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.144463    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"
Sep 17 21:13:39.144892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.144461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" event={"ID":"dc1eedb4-36c8-4533-9558-c7435628e9bc","Type":"ContainerDied","Data":"cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba"}
Sep 17 21:13:39.144892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.144587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk" event={"ID":"dc1eedb4-36c8-4533-9558-c7435628e9bc","Type":"ContainerDied","Data":"ef9f29bc2460e3252052ffada25093338312afc24c34daf91c42002376b72fa7"}
Sep 17 21:13:39.144892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.144620    2577 scope.go:117] "RemoveContainer" containerID="cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba"
Sep 17 21:13:39.155845 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.155823    2577 scope.go:117] "RemoveContainer" containerID="cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba"
Sep 17 21:13:39.156409 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:13:39.156384    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba\": container with ID starting with cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba not found: ID does not exist" containerID="cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba"
Sep 17 21:13:39.156499 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.156418    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba"} err="failed to get container status \"cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba\": rpc error: code = NotFound desc = could not find container \"cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba\": container with ID starting with cb63ca7424d1c85199bc40c6d57d5d1bc3e8fda54ddbc95112bf90cff25933ba not found: ID does not exist"
Sep 17 21:13:39.166228 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.166197    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"]
Sep 17 21:13:39.170936 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.170914    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrhlfk"]
Sep 17 21:13:39.717625 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:39.717592    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" path="/var/lib/kubelet/pods/dc1eedb4-36c8-4533-9558-c7435628e9bc/volumes"
Sep 17 21:13:42.778574 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778542    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"]
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778843    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778854    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778869    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d7ee448-f602-424f-8d8f-745e1e566d99" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778874    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d7ee448-f602-424f-8d8f-745e1e566d99" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778881    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778886    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778960    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d7ee448-f602-424f-8d8f-745e1e566d99" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778969    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b056e6-8c18-40e7-81e1-0b3f0d7cfa56" containerName="main"
Sep 17 21:13:42.779007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.778978    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1eedb4-36c8-4533-9558-c7435628e9bc" containerName="main"
Sep 17 21:13:42.783423 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.783400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:42.787353 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.787327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-l9k6k\""
Sep 17 21:13:42.787476 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.787382    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 21:13:42.787476 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.787338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-grsbv\""
Sep 17 21:13:42.787476 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.787387    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 21:13:42.787476 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.787442    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 21:13:42.791101 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.791077    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"]
Sep 17 21:13:42.905935 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.905898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6h5j\" (UniqueName: \"kubernetes.io/projected/823615e2-e283-40c9-8c62-13482b76e5b8-kube-api-access-c6h5j\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:42.906130 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:42.905984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823615e2-e283-40c9-8c62-13482b76e5b8-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:43.007297 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:43.007225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6h5j\" (UniqueName: \"kubernetes.io/projected/823615e2-e283-40c9-8c62-13482b76e5b8-kube-api-access-c6h5j\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:43.007521 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:43.007360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823615e2-e283-40c9-8c62-13482b76e5b8-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:43.009982 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:43.009955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823615e2-e283-40c9-8c62-13482b76e5b8-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:43.016004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:43.015982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6h5j\" (UniqueName: \"kubernetes.io/projected/823615e2-e283-40c9-8c62-13482b76e5b8-kube-api-access-c6h5j\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:43.095650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:43.095558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:43.372657 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:43.372577    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"]
Sep 17 21:13:43.375885 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:13:43.375850    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823615e2_e283_40c9_8c62_13482b76e5b8.slice/crio-eebf31b7d87957a74429e82377cfc222be2a7927e2a66df11fd1eb1da3a3e5cb WatchSource:0}: Error finding container eebf31b7d87957a74429e82377cfc222be2a7927e2a66df11fd1eb1da3a3e5cb: Status 404 returned error can't find the container with id eebf31b7d87957a74429e82377cfc222be2a7927e2a66df11fd1eb1da3a3e5cb
Sep 17 21:13:44.172946 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:44.172902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" event={"ID":"823615e2-e283-40c9-8c62-13482b76e5b8","Type":"ContainerStarted","Data":"228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777"}
Sep 17 21:13:44.173400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:44.172970    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" event={"ID":"823615e2-e283-40c9-8c62-13482b76e5b8","Type":"ContainerStarted","Data":"eebf31b7d87957a74429e82377cfc222be2a7927e2a66df11fd1eb1da3a3e5cb"}
Sep 17 21:13:44.173400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:44.173139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:13:44.192102 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:13:44.192049    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" podStartSLOduration=2.192032456 podStartE2EDuration="2.192032456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:13:44.189759174 +0000 UTC m=+1463.009051141" watchObservedRunningTime="2026-09-17 21:13:44.192032456 +0000 UTC m=+1463.011324417"
Sep 17 21:14:12.125113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.125076    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"]
Sep 17 21:14:12.128489 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.128472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.130777 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.130754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 21:14:12.130953 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.130928    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 21:14:12.131838 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.131821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qm6fb\""
Sep 17 21:14:12.131953 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.131844    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:14:12.131953 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.131846    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 21:14:12.131953 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.131852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 21:14:12.140609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.140588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"]
Sep 17 21:14:12.246897 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.246860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eec6ae6-a89a-4b05-832a-8686098f88f1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.246897 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.246899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-model-cache\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.247120 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.246930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-home\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.247120 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.247020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-dshm\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.247120 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.247048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.247120 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.247070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eec6ae6-a89a-4b05-832a-8686098f88f1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.247287 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.247145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.247287 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.247180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzjl\" (UniqueName: \"kubernetes.io/projected/0eec6ae6-a89a-4b05-832a-8686098f88f1-kube-api-access-nzzjl\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.347755 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.347708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzjl\" (UniqueName: \"kubernetes.io/projected/0eec6ae6-a89a-4b05-832a-8686098f88f1-kube-api-access-nzzjl\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.347883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eec6ae6-a89a-4b05-832a-8686098f88f1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.347921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-model-cache\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.347956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-home\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348278 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-dshm\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348343 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348343 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eec6ae6-a89a-4b05-832a-8686098f88f1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348434 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348434 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-model-cache\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348674 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348813 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eec6ae6-a89a-4b05-832a-8686098f88f1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.348887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.348857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.349152 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.349127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-home\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.351373 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.351349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eec6ae6-a89a-4b05-832a-8686098f88f1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.351467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.351384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-dshm\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.363467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.363439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzjl\" (UniqueName: \"kubernetes.io/projected/0eec6ae6-a89a-4b05-832a-8686098f88f1-kube-api-access-nzzjl\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-k6qnz\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.397085 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.397006    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"]
Sep 17 21:14:12.400694 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.400669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.403449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.403427    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-hcfdw\""
Sep 17 21:14:12.411355 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.411331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"]
Sep 17 21:14:12.440064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.440021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:12.549890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.549853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hxk\" (UniqueName: \"kubernetes.io/projected/fb0249fe-09e2-45be-86cb-085dfcd54179-kube-api-access-84hxk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.550079 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.549909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0249fe-09e2-45be-86cb-085dfcd54179-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.650968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.650873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hxk\" (UniqueName: \"kubernetes.io/projected/fb0249fe-09e2-45be-86cb-085dfcd54179-kube-api-access-84hxk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.650968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.650957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0249fe-09e2-45be-86cb-085dfcd54179-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.653632 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.653607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0249fe-09e2-45be-86cb-085dfcd54179-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.659485 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.659462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hxk\" (UniqueName: \"kubernetes.io/projected/fb0249fe-09e2-45be-86cb-085dfcd54179-kube-api-access-84hxk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.712002 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.711966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:12.719323 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.719303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"]
Sep 17 21:14:12.722094 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:14:12.722066    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eec6ae6_a89a_4b05_832a_8686098f88f1.slice/crio-eecef2fa38f3b80e3e2d323f82e61527764f88b154c563c55d1a7083530f1cbe WatchSource:0}: Error finding container eecef2fa38f3b80e3e2d323f82e61527764f88b154c563c55d1a7083530f1cbe: Status 404 returned error can't find the container with id eecef2fa38f3b80e3e2d323f82e61527764f88b154c563c55d1a7083530f1cbe
Sep 17 21:14:12.724016 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.723996    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:14:12.988786 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:12.988759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"]
Sep 17 21:14:12.990162 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:14:12.990132    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0249fe_09e2_45be_86cb_085dfcd54179.slice/crio-7121d07e07da5c2b79cc279cfef53508ceb0764629c1c5b8cc7b3309ee65fb4e WatchSource:0}: Error finding container 7121d07e07da5c2b79cc279cfef53508ceb0764629c1c5b8cc7b3309ee65fb4e: Status 404 returned error can't find the container with id 7121d07e07da5c2b79cc279cfef53508ceb0764629c1c5b8cc7b3309ee65fb4e
Sep 17 21:14:13.311789 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:13.311751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" event={"ID":"fb0249fe-09e2-45be-86cb-085dfcd54179","Type":"ContainerStarted","Data":"0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196"}
Sep 17 21:14:13.312270 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:13.311819    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:14:13.312270 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:13.311846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" event={"ID":"fb0249fe-09e2-45be-86cb-085dfcd54179","Type":"ContainerStarted","Data":"7121d07e07da5c2b79cc279cfef53508ceb0764629c1c5b8cc7b3309ee65fb4e"}
Sep 17 21:14:13.313020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:13.312999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" event={"ID":"0eec6ae6-a89a-4b05-832a-8686098f88f1","Type":"ContainerStarted","Data":"eecef2fa38f3b80e3e2d323f82e61527764f88b154c563c55d1a7083530f1cbe"}
Sep 17 21:14:13.331171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:13.331121    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" podStartSLOduration=1.331105554 podStartE2EDuration="1.331105554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:14:13.330462661 +0000 UTC m=+1492.149754636" watchObservedRunningTime="2026-09-17 21:14:13.331105554 +0000 UTC m=+1492.150397515"
Sep 17 21:14:15.181052 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:15.181022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:14:17.338440 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:17.338397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" event={"ID":"0eec6ae6-a89a-4b05-832a-8686098f88f1","Type":"ContainerStarted","Data":"d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa"}
Sep 17 21:14:19.351883 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:19.351849    2577 generic.go:356] "Generic (PLEG): container finished" podID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerID="d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa" exitCode=0
Sep 17 21:14:19.354471 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:19.351929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" event={"ID":"0eec6ae6-a89a-4b05-832a-8686098f88f1","Type":"ContainerDied","Data":"d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa"}
Sep 17 21:14:20.358769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:20.358734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" event={"ID":"0eec6ae6-a89a-4b05-832a-8686098f88f1","Type":"ContainerStarted","Data":"9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426"}
Sep 17 21:14:20.382439 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:20.382394    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" podStartSLOduration=2.404788095 podStartE2EDuration="8.38237939s" totalImagesPullingTime="3.753837629s" totalInitContainerRuntime="2.223753666s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:14:20.380257209 +0000 UTC m=+1499.199549169" watchObservedRunningTime="2026-09-17 21:14:20.38237939 +0000 UTC m=+1499.201671351"
Sep 17 21:14:20.440741 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:20.440711    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:22.440749 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:22.440713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:22.455647 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:22.455623    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:23.384115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:23.384086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:14:44.319883 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:14:44.319850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:15:40.142561 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.142525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"]
Sep 17 21:15:40.154663 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.154631    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"]
Sep 17 21:15:40.164779 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:15:40.164743    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:15:40.164779 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:15:40.164779 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:15:40.164779 ip-10-0-134-35 kubenswrapper[2577]:  > containerID="9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426" cmd=["/bin/sleep","15"]
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.164798    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" message="" err=<
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]:  >
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:15:40.164815    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="main" containerID="cri-o://9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426"
Sep 17 21:15:40.165072 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.164863    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="main" containerID="cri-o://9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426" gracePeriod=60
Sep 17 21:15:40.374039 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.373996    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 21:15:40.514808 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.514787    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:15:40.556600 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556568    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-kserve-provision-location\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556741 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556607    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzzjl\" (UniqueName: \"kubernetes.io/projected/0eec6ae6-a89a-4b05-832a-8686098f88f1-kube-api-access-nzzjl\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556741 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556640    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eec6ae6-a89a-4b05-832a-8686098f88f1-cabundle-cert\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556741 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-tmp-dir\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556741 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-model-cache\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556741 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eec6ae6-a89a-4b05-832a-8686098f88f1-tls-certs\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556999 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-home\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556999 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-dshm\") pod \"0eec6ae6-a89a-4b05-832a-8686098f88f1\" (UID: \"0eec6ae6-a89a-4b05-832a-8686098f88f1\") "
Sep 17 21:15:40.556999 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.556884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.557156 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.557019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.557213 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.557025    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.557721 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.557118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-home" (OuterVolumeSpecName: "home") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.557721 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.557646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eec6ae6-a89a-4b05-832a-8686098f88f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:15:40.559273 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.559249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eec6ae6-a89a-4b05-832a-8686098f88f1-kube-api-access-nzzjl" (OuterVolumeSpecName: "kube-api-access-nzzjl") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "kube-api-access-nzzjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:15:40.559273 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.559229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-dshm" (OuterVolumeSpecName: "dshm") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.559484 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.559464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eec6ae6-a89a-4b05-832a-8686098f88f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:15:40.576568 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.576544    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0eec6ae6-a89a-4b05-832a-8686098f88f1" (UID: "0eec6ae6-a89a-4b05-832a-8686098f88f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.658062 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.658032    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.658062 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.658061    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eec6ae6-a89a-4b05-832a-8686098f88f1-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.658062 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.658069    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.658293 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.658077    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.658293 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.658085    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eec6ae6-a89a-4b05-832a-8686098f88f1-kserve-provision-location\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.658293 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.658095    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzjl\" (UniqueName: \"kubernetes.io/projected/0eec6ae6-a89a-4b05-832a-8686098f88f1-kube-api-access-nzzjl\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.658293 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.658103    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eec6ae6-a89a-4b05-832a-8686098f88f1-cabundle-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.760232 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.760199    2577 generic.go:356] "Generic (PLEG): container finished" podID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerID="9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426" exitCode=0
Sep 17 21:15:40.760431 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.760267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" event={"ID":"0eec6ae6-a89a-4b05-832a-8686098f88f1","Type":"ContainerDied","Data":"9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426"}
Sep 17 21:15:40.760431 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.760279    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"
Sep 17 21:15:40.760431 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.760312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz" event={"ID":"0eec6ae6-a89a-4b05-832a-8686098f88f1","Type":"ContainerDied","Data":"eecef2fa38f3b80e3e2d323f82e61527764f88b154c563c55d1a7083530f1cbe"}
Sep 17 21:15:40.760431 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.760332    2577 scope.go:117] "RemoveContainer" containerID="9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426"
Sep 17 21:15:40.770412 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.770393    2577 scope.go:117] "RemoveContainer" containerID="d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa"
Sep 17 21:15:40.778009 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.777995    2577 scope.go:117] "RemoveContainer" containerID="9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426"
Sep 17 21:15:40.778262 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:15:40.778222    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426\": container with ID starting with 9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426 not found: ID does not exist" containerID="9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426"
Sep 17 21:15:40.778321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.778275    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426"} err="failed to get container status \"9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426\": rpc error: code = NotFound desc = could not find container \"9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426\": container with ID starting with 9e7f971f0094269a12df33a4ad1da871b694ad02f8493fad9a7c11c0c5461426 not found: ID does not exist"
Sep 17 21:15:40.778321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.778298    2577 scope.go:117] "RemoveContainer" containerID="d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa"
Sep 17 21:15:40.778563 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:15:40.778546    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa\": container with ID starting with d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa not found: ID does not exist" containerID="d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa"
Sep 17 21:15:40.778613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.778567    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa"} err="failed to get container status \"d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa\": rpc error: code = NotFound desc = could not find container \"d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa\": container with ID starting with d15cc4c3cb388f4a2b8832b5a566626506427752aca0136f55fcf4a9b6274baa not found: ID does not exist"
Sep 17 21:15:40.782578 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.782558    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"]
Sep 17 21:15:40.786694 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:40.786673    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-k6qnz"]
Sep 17 21:15:41.718102 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:41.718064    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" path="/var/lib/kubelet/pods/0eec6ae6-a89a-4b05-832a-8686098f88f1/volumes"
Sep 17 21:15:44.320704 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:44.320656    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:15:44.591817 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:44.591732    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"]
Sep 17 21:15:45.181154 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:45.181117    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:15:54.320530 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:54.320492    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:15:55.164407 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:55.164368    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main" containerID="cri-o://0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196" gracePeriod=45
Sep 17 21:15:55.181097 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:55.181059    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:15:59.610794 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:15:59.610752    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main" containerID="cri-o://228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777" gracePeriod=45
Sep 17 21:16:04.320432 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:04.320389    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:16:04.320894 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:04.320557    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:16:05.181499 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:05.181461    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:16:05.181747 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:05.181573    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:16:14.320648 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:14.320552    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:16:15.181601 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:15.181558    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:16:24.320766 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:24.320726    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:16:25.181338 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.181308    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:16:25.417940 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.417914    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:16:25.530143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.530114    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84hxk\" (UniqueName: \"kubernetes.io/projected/fb0249fe-09e2-45be-86cb-085dfcd54179-kube-api-access-84hxk\") pod \"fb0249fe-09e2-45be-86cb-085dfcd54179\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") "
Sep 17 21:16:25.530358 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.530158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0249fe-09e2-45be-86cb-085dfcd54179-tls-certs\") pod \"fb0249fe-09e2-45be-86cb-085dfcd54179\" (UID: \"fb0249fe-09e2-45be-86cb-085dfcd54179\") "
Sep 17 21:16:25.532431 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.532405    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0249fe-09e2-45be-86cb-085dfcd54179-kube-api-access-84hxk" (OuterVolumeSpecName: "kube-api-access-84hxk") pod "fb0249fe-09e2-45be-86cb-085dfcd54179" (UID: "fb0249fe-09e2-45be-86cb-085dfcd54179"). InnerVolumeSpecName "kube-api-access-84hxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:16:25.532533 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.532448    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0249fe-09e2-45be-86cb-085dfcd54179-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb0249fe-09e2-45be-86cb-085dfcd54179" (UID: "fb0249fe-09e2-45be-86cb-085dfcd54179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:16:25.631503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.631469    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hxk\" (UniqueName: \"kubernetes.io/projected/fb0249fe-09e2-45be-86cb-085dfcd54179-kube-api-access-84hxk\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:16:25.631503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.631496    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0249fe-09e2-45be-86cb-085dfcd54179-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:16:25.990101 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.990007    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerID="0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196" exitCode=0
Sep 17 21:16:25.990101 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.990061    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"
Sep 17 21:16:25.990101 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.990081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" event={"ID":"fb0249fe-09e2-45be-86cb-085dfcd54179","Type":"ContainerDied","Data":"0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196"}
Sep 17 21:16:25.990398 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.990143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv" event={"ID":"fb0249fe-09e2-45be-86cb-085dfcd54179","Type":"ContainerDied","Data":"7121d07e07da5c2b79cc279cfef53508ceb0764629c1c5b8cc7b3309ee65fb4e"}
Sep 17 21:16:25.990398 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:25.990164    2577 scope.go:117] "RemoveContainer" containerID="0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196"
Sep 17 21:16:26.001687 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:26.001659    2577 scope.go:117] "RemoveContainer" containerID="0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196"
Sep 17 21:16:26.002111 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:16:26.002081    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196\": container with ID starting with 0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196 not found: ID does not exist" containerID="0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196"
Sep 17 21:16:26.002197 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:26.002127    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196"} err="failed to get container status \"0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196\": rpc error: code = NotFound desc = could not find container \"0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196\": container with ID starting with 0f079fa613965524823f6109742fda7ad765c7d3f95191b379d5fddc9a909196 not found: ID does not exist"
Sep 17 21:16:26.007411 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:26.007389    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"]
Sep 17 21:16:26.012952 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:26.012928    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69djpsv"]
Sep 17 21:16:27.718332 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:27.718303    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" path="/var/lib/kubelet/pods/fb0249fe-09e2-45be-86cb-085dfcd54179/volumes"
Sep 17 21:16:29.865861 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:29.865837    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:16:29.963188 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:29.963093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6h5j\" (UniqueName: \"kubernetes.io/projected/823615e2-e283-40c9-8c62-13482b76e5b8-kube-api-access-c6h5j\") pod \"823615e2-e283-40c9-8c62-13482b76e5b8\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") "
Sep 17 21:16:29.963188 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:29.963151    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823615e2-e283-40c9-8c62-13482b76e5b8-tls-certs\") pod \"823615e2-e283-40c9-8c62-13482b76e5b8\" (UID: \"823615e2-e283-40c9-8c62-13482b76e5b8\") "
Sep 17 21:16:29.965475 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:29.965438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823615e2-e283-40c9-8c62-13482b76e5b8-kube-api-access-c6h5j" (OuterVolumeSpecName: "kube-api-access-c6h5j") pod "823615e2-e283-40c9-8c62-13482b76e5b8" (UID: "823615e2-e283-40c9-8c62-13482b76e5b8"). InnerVolumeSpecName "kube-api-access-c6h5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:16:29.965593 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:29.965503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823615e2-e283-40c9-8c62-13482b76e5b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "823615e2-e283-40c9-8c62-13482b76e5b8" (UID: "823615e2-e283-40c9-8c62-13482b76e5b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:16:30.010992 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.010962    2577 generic.go:356] "Generic (PLEG): container finished" podID="823615e2-e283-40c9-8c62-13482b76e5b8" containerID="228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777" exitCode=0
Sep 17 21:16:30.011108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.011001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" event={"ID":"823615e2-e283-40c9-8c62-13482b76e5b8","Type":"ContainerDied","Data":"228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777"}
Sep 17 21:16:30.011108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.011012    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"
Sep 17 21:16:30.011108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.011055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz" event={"ID":"823615e2-e283-40c9-8c62-13482b76e5b8","Type":"ContainerDied","Data":"eebf31b7d87957a74429e82377cfc222be2a7927e2a66df11fd1eb1da3a3e5cb"}
Sep 17 21:16:30.011108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.011087    2577 scope.go:117] "RemoveContainer" containerID="228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777"
Sep 17 21:16:30.019761 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.019743    2577 scope.go:117] "RemoveContainer" containerID="228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777"
Sep 17 21:16:30.020029 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:16:30.020012    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777\": container with ID starting with 228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777 not found: ID does not exist" containerID="228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777"
Sep 17 21:16:30.020077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.020035    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777"} err="failed to get container status \"228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777\": rpc error: code = NotFound desc = could not find container \"228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777\": container with ID starting with 228b51d6a0de175088715dded2a48d5f2aa0f0248725605729164976af3d9777 not found: ID does not exist"
Sep 17 21:16:30.032031 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.031993    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"]
Sep 17 21:16:30.034871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.034849    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwd5cz"]
Sep 17 21:16:30.064445 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.064427    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6h5j\" (UniqueName: \"kubernetes.io/projected/823615e2-e283-40c9-8c62-13482b76e5b8-kube-api-access-c6h5j\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:16:30.064531 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:30.064447    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823615e2-e283-40c9-8c62-13482b76e5b8-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:16:31.717316 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:31.717286    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" path="/var/lib/kubelet/pods/823615e2-e283-40c9-8c62-13482b76e5b8/volumes"
Sep 17 21:16:37.116773 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.116738    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"]
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117032    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117043    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117055    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117061    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117074    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117081    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117090    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="storage-initializer"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117095    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="storage-initializer"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117142    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="823615e2-e283-40c9-8c62-13482b76e5b8" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117148    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0249fe-09e2-45be-86cb-085dfcd54179" containerName="main"
Sep 17 21:16:37.117167 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.117154    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eec6ae6-a89a-4b05-832a-8686098f88f1" containerName="main"
Sep 17 21:16:37.121738 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.121715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124416 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-home\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124549 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124549 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124549 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124707 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124707 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124707 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-dshm\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124707 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5c2p\" (UniqueName: \"kubernetes.io/projected/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kube-api-access-s5c2p\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.124919 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124789    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 21:16:37.124919 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 21:16:37.124919 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.124847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 21:16:37.125421 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.125406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:16:37.125626 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.125612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 21:16:37.125672 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.125642    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-bh28s\""
Sep 17 21:16:37.140727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.140697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"]
Sep 17 21:16:37.225304 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225304 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-dshm\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5c2p\" (UniqueName: \"kubernetes.io/projected/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kube-api-access-s5c2p\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-home\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225840 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225904 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.225948 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.226012 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.225988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-home\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.226117 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.226090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.228284 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.228259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-dshm\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.228397 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.228382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.235415 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.235393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5c2p\" (UniqueName: \"kubernetes.io/projected/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kube-api-access-s5c2p\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-6fbp5\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.431925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.431834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:37.452639 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.452611    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"]
Sep 17 21:16:37.456184 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.456163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:37.460014 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.459993    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-w45bx\""
Sep 17 21:16:37.473630 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.473602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"]
Sep 17 21:16:37.527278 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.527230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea794d1b-afe6-4590-9235-35d536816f80-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:37.527467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.527295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4d7j\" (UniqueName: \"kubernetes.io/projected/ea794d1b-afe6-4590-9235-35d536816f80-kube-api-access-t4d7j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:37.627788 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.627755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea794d1b-afe6-4590-9235-35d536816f80-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:37.627943 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.627796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4d7j\" (UniqueName: \"kubernetes.io/projected/ea794d1b-afe6-4590-9235-35d536816f80-kube-api-access-t4d7j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:37.630316 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.630296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea794d1b-afe6-4590-9235-35d536816f80-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:37.636293 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.636269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4d7j\" (UniqueName: \"kubernetes.io/projected/ea794d1b-afe6-4590-9235-35d536816f80-kube-api-access-t4d7j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:37.718465 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.718437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"]
Sep 17 21:16:37.719683 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:16:37.719661    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2000b9_2c1c_4bbc_a35f_64b3bf91f271.slice/crio-923edd21fff6c57e39dab53adf48e2ff475a46914be0a8f11b27f7e63c771933 WatchSource:0}: Error finding container 923edd21fff6c57e39dab53adf48e2ff475a46914be0a8f11b27f7e63c771933: Status 404 returned error can't find the container with id 923edd21fff6c57e39dab53adf48e2ff475a46914be0a8f11b27f7e63c771933
Sep 17 21:16:37.769859 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:37.769833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:38.044307 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:38.044268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"]
Sep 17 21:16:38.046780 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:16:38.046749    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea794d1b_afe6_4590_9235_35d536816f80.slice/crio-8e671e9d2c9e02a3581514461664079b2311506e7f4e3fa429f8cd9dfa6b1ce5 WatchSource:0}: Error finding container 8e671e9d2c9e02a3581514461664079b2311506e7f4e3fa429f8cd9dfa6b1ce5: Status 404 returned error can't find the container with id 8e671e9d2c9e02a3581514461664079b2311506e7f4e3fa429f8cd9dfa6b1ce5
Sep 17 21:16:38.051967 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:38.051938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" event={"ID":"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271","Type":"ContainerStarted","Data":"3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01"}
Sep 17 21:16:38.052059 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:38.051992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" event={"ID":"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271","Type":"ContainerStarted","Data":"923edd21fff6c57e39dab53adf48e2ff475a46914be0a8f11b27f7e63c771933"}
Sep 17 21:16:39.060855 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:39.060801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" event={"ID":"ea794d1b-afe6-4590-9235-35d536816f80","Type":"ContainerStarted","Data":"c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727"}
Sep 17 21:16:39.061505 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:39.061476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:16:39.061811 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:39.061521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" event={"ID":"ea794d1b-afe6-4590-9235-35d536816f80","Type":"ContainerStarted","Data":"8e671e9d2c9e02a3581514461664079b2311506e7f4e3fa429f8cd9dfa6b1ce5"}
Sep 17 21:16:39.087011 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:39.086960    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" podStartSLOduration=2.086946091 podStartE2EDuration="2.086946091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:16:39.085492002 +0000 UTC m=+1637.904783964" watchObservedRunningTime="2026-09-17 21:16:39.086946091 +0000 UTC m=+1637.906238053"
Sep 17 21:16:40.067599 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:40.067566    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerID="3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01" exitCode=0
Sep 17 21:16:40.067938 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:40.067635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" event={"ID":"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271","Type":"ContainerDied","Data":"3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01"}
Sep 17 21:16:41.074081 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:41.074041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" event={"ID":"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271","Type":"ContainerStarted","Data":"693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09"}
Sep 17 21:16:41.098001 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:41.097947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" podStartSLOduration=1.859782616 podStartE2EDuration="4.097933092s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.238150476s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:16:41.096205662 +0000 UTC m=+1639.915497623" watchObservedRunningTime="2026-09-17 21:16:41.097933092 +0000 UTC m=+1639.917225053"
Sep 17 21:16:41.432459 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:41.432371    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:47.432796 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:47.432758    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:47.445882 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:47.445857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:48.121296 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:48.121269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:16:55.995361 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:55.995328    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"]
Sep 17 21:16:55.998784 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:55.998757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.001396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.001367    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 21:16:56.002913 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.002890    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 21:16:56.003072 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.003052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-9twgc\""
Sep 17 21:16:56.003157 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.003142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-2znt5\""
Sep 17 21:16:56.003223 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.003162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 21:16:56.010030 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.010006    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"]
Sep 17 21:16:56.096857 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.096817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29gns\" (UniqueName: \"kubernetes.io/projected/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-kube-api-access-29gns\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.097035 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.096870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.197959 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.197930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29gns\" (UniqueName: \"kubernetes.io/projected/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-kube-api-access-29gns\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.198143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.197968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.200657 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.200624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.206626 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.206597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29gns\" (UniqueName: \"kubernetes.io/projected/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-kube-api-access-29gns\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.310216 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.310177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:56.586914 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:56.586877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"]
Sep 17 21:16:56.589076 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:16:56.589052    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4493e6_0a32_4033_8ede_de6fb2ac7a8d.slice/crio-ab1334f06778fe18cbd7d7a9c85a03cfcbdee5159a8eb79bcb19bf17fe13e38c WatchSource:0}: Error finding container ab1334f06778fe18cbd7d7a9c85a03cfcbdee5159a8eb79bcb19bf17fe13e38c: Status 404 returned error can't find the container with id ab1334f06778fe18cbd7d7a9c85a03cfcbdee5159a8eb79bcb19bf17fe13e38c
Sep 17 21:16:57.156942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:57.156905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" event={"ID":"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d","Type":"ContainerStarted","Data":"c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f"}
Sep 17 21:16:57.157389 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:57.156960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" event={"ID":"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d","Type":"ContainerStarted","Data":"ab1334f06778fe18cbd7d7a9c85a03cfcbdee5159a8eb79bcb19bf17fe13e38c"}
Sep 17 21:16:57.157389 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:57.157006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:16:57.179384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:16:57.179333    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" podStartSLOduration=2.179318501 podStartE2EDuration="2.179318501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:16:57.17742552 +0000 UTC m=+1655.996717502" watchObservedRunningTime="2026-09-17 21:16:57.179318501 +0000 UTC m=+1655.998610463"
Sep 17 21:17:11.076443 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:11.076396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:17:14.791695 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:14.791652    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"]
Sep 17 21:17:14.793679 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:14.793649    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"]
Sep 17 21:17:14.801875 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:17:14.801843    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:17:14.801875 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:17:14.801875 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:17:14.801875 ip-10-0-134-35 kubenswrapper[2577]:  > containerID="693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09" cmd=["/bin/sleep","15"]
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:14.801897    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" message="" err=<
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]:  >
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:17:14.801913    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="main" containerID="cri-o://693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09"
Sep 17 21:17:14.802146 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:14.801955    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="main" containerID="cri-o://693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09" gracePeriod=60
Sep 17 21:17:15.165465 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.165441    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:17:15.253399 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.253364    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerID="693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09" exitCode=0
Sep 17 21:17:15.253573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.253414    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"
Sep 17 21:17:15.253573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.253451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" event={"ID":"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271","Type":"ContainerDied","Data":"693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09"}
Sep 17 21:17:15.253573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.253504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" event={"ID":"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271","Type":"ContainerDied","Data":"923edd21fff6c57e39dab53adf48e2ff475a46914be0a8f11b27f7e63c771933"}
Sep 17 21:17:15.253573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.253528    2577 scope.go:117] "RemoveContainer" containerID="693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09"
Sep 17 21:17:15.258980 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.258959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-model-cache\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259094 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259002    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5c2p\" (UniqueName: \"kubernetes.io/projected/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kube-api-access-s5c2p\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259094 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-home\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259094 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tls-certs\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259225 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-dshm\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259225 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tmp-dir\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259225 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259160    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kserve-provision-location\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259399 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-cabundle-cert\") pod \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\" (UID: \"1c2000b9-2c1c-4bbc-a35f-64b3bf91f271\") "
Sep 17 21:17:15.259399 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259261    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.259399 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-home" (OuterVolumeSpecName: "home") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.259575 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259558    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.259636 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259577    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.259744 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.259882 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.259861    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:17:15.261753 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.261724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kube-api-access-s5c2p" (OuterVolumeSpecName: "kube-api-access-s5c2p") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "kube-api-access-s5c2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:17:15.261961 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.261935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-dshm" (OuterVolumeSpecName: "dshm") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.262145 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.262127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:17:15.263020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.262999    2577 scope.go:117] "RemoveContainer" containerID="3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01"
Sep 17 21:17:15.276621 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.276605    2577 scope.go:117] "RemoveContainer" containerID="693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09"
Sep 17 21:17:15.276943 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:17:15.276918    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09\": container with ID starting with 693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09 not found: ID does not exist" containerID="693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09"
Sep 17 21:17:15.277018 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.276954    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09"} err="failed to get container status \"693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09\": rpc error: code = NotFound desc = could not find container \"693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09\": container with ID starting with 693ae9db9a3dd7cb884d80bca76e363d1e2dbbc44a6e8866287e7108b0516e09 not found: ID does not exist"
Sep 17 21:17:15.277018 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.276982    2577 scope.go:117] "RemoveContainer" containerID="3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01"
Sep 17 21:17:15.277294 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:17:15.277273    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01\": container with ID starting with 3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01 not found: ID does not exist" containerID="3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01"
Sep 17 21:17:15.277374 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.277302    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01"} err="failed to get container status \"3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01\": rpc error: code = NotFound desc = could not find container \"3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01\": container with ID starting with 3021f415b2f4d4d882d9d1f230629389c0aea22754a6aebe5edb545959a5ca01 not found: ID does not exist"
Sep 17 21:17:15.280615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.280590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" (UID: "1c2000b9-2c1c-4bbc-a35f-64b3bf91f271"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.361004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.360935    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5c2p\" (UniqueName: \"kubernetes.io/projected/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kube-api-access-s5c2p\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.361004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.360965    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.361004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.360976    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.361004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.360984    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.361004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.360991    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-kserve-provision-location\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.361004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.360999    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271-cabundle-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.579398 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.579341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"]
Sep 17 21:17:15.581068 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.581049    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5"]
Sep 17 21:17:15.717453 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:15.717376    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" path="/var/lib/kubelet/pods/1c2000b9-2c1c-4bbc-a35f-64b3bf91f271/volumes"
Sep 17 21:17:16.111687 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:16.111645    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-6fbp5" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Sep 17 21:17:21.075834 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:21.075797    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:17:28.164283 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:28.164229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:17:29.813688 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:29.813645    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main" containerID="cri-o://c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727" gracePeriod=45
Sep 17 21:17:31.076080 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:31.076041    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:17:41.075729 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:41.075692    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:17:41.076105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:41.075805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:17:51.076339 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:17:51.076299    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:18:00.063478 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.063452    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:18:00.135001 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.134936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea794d1b-afe6-4590-9235-35d536816f80-tls-certs\") pod \"ea794d1b-afe6-4590-9235-35d536816f80\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") "
Sep 17 21:18:00.135001 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.134971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4d7j\" (UniqueName: \"kubernetes.io/projected/ea794d1b-afe6-4590-9235-35d536816f80-kube-api-access-t4d7j\") pod \"ea794d1b-afe6-4590-9235-35d536816f80\" (UID: \"ea794d1b-afe6-4590-9235-35d536816f80\") "
Sep 17 21:18:00.137091 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.137059    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea794d1b-afe6-4590-9235-35d536816f80-kube-api-access-t4d7j" (OuterVolumeSpecName: "kube-api-access-t4d7j") pod "ea794d1b-afe6-4590-9235-35d536816f80" (UID: "ea794d1b-afe6-4590-9235-35d536816f80"). InnerVolumeSpecName "kube-api-access-t4d7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:18:00.137192 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.137066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea794d1b-afe6-4590-9235-35d536816f80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea794d1b-afe6-4590-9235-35d536816f80" (UID: "ea794d1b-afe6-4590-9235-35d536816f80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:18:00.235421 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.235394    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea794d1b-afe6-4590-9235-35d536816f80-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:18:00.235421 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.235416    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4d7j\" (UniqueName: \"kubernetes.io/projected/ea794d1b-afe6-4590-9235-35d536816f80-kube-api-access-t4d7j\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:18:00.478285 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.478170    2577 generic.go:356] "Generic (PLEG): container finished" podID="ea794d1b-afe6-4590-9235-35d536816f80" containerID="c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727" exitCode=0
Sep 17 21:18:00.478285 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.478219    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"
Sep 17 21:18:00.478524 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.478285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" event={"ID":"ea794d1b-afe6-4590-9235-35d536816f80","Type":"ContainerDied","Data":"c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727"}
Sep 17 21:18:00.478524 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.478352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc" event={"ID":"ea794d1b-afe6-4590-9235-35d536816f80","Type":"ContainerDied","Data":"8e671e9d2c9e02a3581514461664079b2311506e7f4e3fa429f8cd9dfa6b1ce5"}
Sep 17 21:18:00.478524 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.478382    2577 scope.go:117] "RemoveContainer" containerID="c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727"
Sep 17 21:18:00.487221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.487206    2577 scope.go:117] "RemoveContainer" containerID="c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727"
Sep 17 21:18:00.487528 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:18:00.487512    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727\": container with ID starting with c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727 not found: ID does not exist" containerID="c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727"
Sep 17 21:18:00.487577 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.487534    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727"} err="failed to get container status \"c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727\": rpc error: code = NotFound desc = could not find container \"c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727\": container with ID starting with c270a1de215599ec289db7239b601753845124bf100ca8096c1d10b08dae5727 not found: ID does not exist"
Sep 17 21:18:00.506930 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.506899    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"]
Sep 17 21:18:00.508571 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:00.508551    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qktvc"]
Sep 17 21:18:01.718100 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:01.718064    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea794d1b-afe6-4590-9235-35d536816f80" path="/var/lib/kubelet/pods/ea794d1b-afe6-4590-9235-35d536816f80/volumes"
Sep 17 21:18:18.911500 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911468    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"]
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911766    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="storage-initializer"
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911777    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="storage-initializer"
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911789    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main"
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911795    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main"
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="main"
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911807    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="main"
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911849    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2000b9-2c1c-4bbc-a35f-64b3bf91f271" containerName="main"
Sep 17 21:18:18.911871 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.911858    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea794d1b-afe6-4590-9235-35d536816f80" containerName="main"
Sep 17 21:18:18.916080 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.916060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:18.918847 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.918825    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 21:18:18.919763 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.919741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 21:18:18.919763 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.919759    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-c7fn6\""
Sep 17 21:18:18.919927 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.919749    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7lbzg\""
Sep 17 21:18:18.919927 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.919817    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 21:18:18.935670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.935645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"]
Sep 17 21:18:18.987728 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.987684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwlg\" (UniqueName: \"kubernetes.io/projected/467147a7-aa64-41d7-ae20-b4a250bb7fff-kube-api-access-lcwlg\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:18.988015 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:18.987982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467147a7-aa64-41d7-ae20-b4a250bb7fff-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:19.088560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:19.088524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwlg\" (UniqueName: \"kubernetes.io/projected/467147a7-aa64-41d7-ae20-b4a250bb7fff-kube-api-access-lcwlg\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:19.088779 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:19.088572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467147a7-aa64-41d7-ae20-b4a250bb7fff-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:19.091283 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:19.091259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467147a7-aa64-41d7-ae20-b4a250bb7fff-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:19.098566 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:19.098544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwlg\" (UniqueName: \"kubernetes.io/projected/467147a7-aa64-41d7-ae20-b4a250bb7fff-kube-api-access-lcwlg\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:19.226692 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:19.226594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:19.511700 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:19.511666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"]
Sep 17 21:18:19.514940 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:18:19.514910    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467147a7_aa64_41d7_ae20_b4a250bb7fff.slice/crio-7d9b9930b1e18d9b0f7af2ccfd242c5a49847bb1a79592c62bd72ac047814f10 WatchSource:0}: Error finding container 7d9b9930b1e18d9b0f7af2ccfd242c5a49847bb1a79592c62bd72ac047814f10: Status 404 returned error can't find the container with id 7d9b9930b1e18d9b0f7af2ccfd242c5a49847bb1a79592c62bd72ac047814f10
Sep 17 21:18:19.577448 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:19.577408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" event={"ID":"467147a7-aa64-41d7-ae20-b4a250bb7fff","Type":"ContainerStarted","Data":"7d9b9930b1e18d9b0f7af2ccfd242c5a49847bb1a79592c62bd72ac047814f10"}
Sep 17 21:18:20.583949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:20.583907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" event={"ID":"467147a7-aa64-41d7-ae20-b4a250bb7fff","Type":"ContainerStarted","Data":"6d8c06ac1d0cce6ee35729bd347247e5e99d324b279fbecd7416a582ce9f8d94"}
Sep 17 21:18:20.584377 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:20.583997    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:20.604314 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:20.604263    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" podStartSLOduration=2.604230604 podStartE2EDuration="2.604230604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:18:20.602162348 +0000 UTC m=+1739.421454324" watchObservedRunningTime="2026-09-17 21:18:20.604230604 +0000 UTC m=+1739.423522566"
Sep 17 21:18:51.591960 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:51.591927    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:18:57.768964 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:18:57.768932    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"]
Sep 17 21:19:01.592594 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:01.592552    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:01.619100 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:01.619064    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"]
Sep 17 21:19:08.163999 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:08.163954    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:11.592070 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:11.592026    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:12.789103 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:12.789062    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main" containerID="cri-o://6d8c06ac1d0cce6ee35729bd347247e5e99d324b279fbecd7416a582ce9f8d94" gracePeriod=45
Sep 17 21:19:16.639029 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:16.638988    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main" containerID="cri-o://c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f" gracePeriod=45
Sep 17 21:19:18.163832 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:18.163786    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:21.592254 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:21.592209    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:21.592674 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:21.592337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:19:28.164166 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:28.164126    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:28.164749 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:28.164224    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:19:31.592610 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:31.592564    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:38.164677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:38.164634    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:41.592212 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:41.592167    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:19:42.981400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:42.981355    2577 generic.go:356] "Generic (PLEG): container finished" podID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerID="6d8c06ac1d0cce6ee35729bd347247e5e99d324b279fbecd7416a582ce9f8d94" exitCode=0
Sep 17 21:19:42.981867 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:42.981450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" event={"ID":"467147a7-aa64-41d7-ae20-b4a250bb7fff","Type":"ContainerDied","Data":"6d8c06ac1d0cce6ee35729bd347247e5e99d324b279fbecd7416a582ce9f8d94"}
Sep 17 21:19:43.047548 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.047489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:19:43.074817 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.074789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467147a7-aa64-41d7-ae20-b4a250bb7fff-tls-certs\") pod \"467147a7-aa64-41d7-ae20-b4a250bb7fff\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") "
Sep 17 21:19:43.074817 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.074822    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcwlg\" (UniqueName: \"kubernetes.io/projected/467147a7-aa64-41d7-ae20-b4a250bb7fff-kube-api-access-lcwlg\") pod \"467147a7-aa64-41d7-ae20-b4a250bb7fff\" (UID: \"467147a7-aa64-41d7-ae20-b4a250bb7fff\") "
Sep 17 21:19:43.077212 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.077179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467147a7-aa64-41d7-ae20-b4a250bb7fff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "467147a7-aa64-41d7-ae20-b4a250bb7fff" (UID: "467147a7-aa64-41d7-ae20-b4a250bb7fff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:19:43.077348 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.077187    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467147a7-aa64-41d7-ae20-b4a250bb7fff-kube-api-access-lcwlg" (OuterVolumeSpecName: "kube-api-access-lcwlg") pod "467147a7-aa64-41d7-ae20-b4a250bb7fff" (UID: "467147a7-aa64-41d7-ae20-b4a250bb7fff"). InnerVolumeSpecName "kube-api-access-lcwlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:19:43.176099 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.176042    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467147a7-aa64-41d7-ae20-b4a250bb7fff-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:19:43.176099 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.176112    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcwlg\" (UniqueName: \"kubernetes.io/projected/467147a7-aa64-41d7-ae20-b4a250bb7fff-kube-api-access-lcwlg\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:19:43.988691 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.988605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh" event={"ID":"467147a7-aa64-41d7-ae20-b4a250bb7fff","Type":"ContainerDied","Data":"7d9b9930b1e18d9b0f7af2ccfd242c5a49847bb1a79592c62bd72ac047814f10"}
Sep 17 21:19:43.988691 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.988662    2577 scope.go:117] "RemoveContainer" containerID="6d8c06ac1d0cce6ee35729bd347247e5e99d324b279fbecd7416a582ce9f8d94"
Sep 17 21:19:43.989188 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:43.988617    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"
Sep 17 21:19:44.010158 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:44.010126    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"]
Sep 17 21:19:44.013633 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:44.013609    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dq7xsh"]
Sep 17 21:19:45.717385 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:45.717350    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" path="/var/lib/kubelet/pods/467147a7-aa64-41d7-ae20-b4a250bb7fff/volumes"
Sep 17 21:19:46.895685 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:46.895659    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:19:47.005032 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.004941    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-tls-certs\") pod \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") "
Sep 17 21:19:47.005032 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.004992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29gns\" (UniqueName: \"kubernetes.io/projected/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-kube-api-access-29gns\") pod \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\" (UID: \"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d\") "
Sep 17 21:19:47.005874 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.005840    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerID="c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f" exitCode=0
Sep 17 21:19:47.005995 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.005902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" event={"ID":"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d","Type":"ContainerDied","Data":"c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f"}
Sep 17 21:19:47.005995 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.005939    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"
Sep 17 21:19:47.005995 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.005961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb" event={"ID":"cd4493e6-0a32-4033-8ede-de6fb2ac7a8d","Type":"ContainerDied","Data":"ab1334f06778fe18cbd7d7a9c85a03cfcbdee5159a8eb79bcb19bf17fe13e38c"}
Sep 17 21:19:47.006113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.005995    2577 scope.go:117] "RemoveContainer" containerID="c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f"
Sep 17 21:19:47.007622 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.007598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-kube-api-access-29gns" (OuterVolumeSpecName: "kube-api-access-29gns") pod "cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" (UID: "cd4493e6-0a32-4033-8ede-de6fb2ac7a8d"). InnerVolumeSpecName "kube-api-access-29gns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:19:47.007720 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.007692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" (UID: "cd4493e6-0a32-4033-8ede-de6fb2ac7a8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:19:47.017467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.017449    2577 scope.go:117] "RemoveContainer" containerID="c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f"
Sep 17 21:19:47.017750 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:19:47.017717    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f\": container with ID starting with c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f not found: ID does not exist" containerID="c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f"
Sep 17 21:19:47.017825 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.017755    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f"} err="failed to get container status \"c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f\": rpc error: code = NotFound desc = could not find container \"c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f\": container with ID starting with c3316c601a970398ed335df1541fde9d0a341b1c4a8581c25d50954c8d404e8f not found: ID does not exist"
Sep 17 21:19:47.106324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.106293    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:19:47.106324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.106321    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29gns\" (UniqueName: \"kubernetes.io/projected/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d-kube-api-access-29gns\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:19:47.338694 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.338659    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"]
Sep 17 21:19:47.349873 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.349842    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-27jnb"]
Sep 17 21:19:47.717767 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:47.717685    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" path="/var/lib/kubelet/pods/cd4493e6-0a32-4033-8ede-de6fb2ac7a8d/volumes"
Sep 17 21:19:52.126299 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.126265    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"]
Sep 17 21:19:52.126645 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.126584    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main"
Sep 17 21:19:52.126645 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.126598    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main"
Sep 17 21:19:52.126645 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.126611    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main"
Sep 17 21:19:52.126645 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.126617    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main"
Sep 17 21:19:52.126789 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.126685    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="467147a7-aa64-41d7-ae20-b4a250bb7fff" containerName="main"
Sep 17 21:19:52.126789 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.126696    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd4493e6-0a32-4033-8ede-de6fb2ac7a8d" containerName="main"
Sep 17 21:19:52.130377 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.130359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.135098 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.135073    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 21:19:52.139825 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.139804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-gz6c2\""
Sep 17 21:19:52.139928 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.139842    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 21:19:52.139928 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.139858    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 21:19:52.162430 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.162398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"]
Sep 17 21:19:52.247483 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247483 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54wp\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-kube-api-access-g54wp\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.247754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.247755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348456 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348456 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348718 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348718 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54wp\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-kube-api-access-g54wp\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348718 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348718 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348938 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348938 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348938 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.348938 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.349121 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.348983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.349367 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.349341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.349367 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.349362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.349692 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.349394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.351226 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.351201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.351437 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.351419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.361996 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.361972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54wp\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-kube-api-access-g54wp\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.367145 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.367093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-546nb\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.440367 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.440339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:52.718115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.718088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"]
Sep 17 21:19:52.720497 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:19:52.720472    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330bdcf1_42c4_4d07_9cbf_6ffc203b21c2.slice/crio-4192aae4da68b0b2061d2aee9064af512d9ba415d0de44ad399b0cfeef245a07 WatchSource:0}: Error finding container 4192aae4da68b0b2061d2aee9064af512d9ba415d0de44ad399b0cfeef245a07: Status 404 returned error can't find the container with id 4192aae4da68b0b2061d2aee9064af512d9ba415d0de44ad399b0cfeef245a07
Sep 17 21:19:52.722451 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.722421    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:19:52.722548 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.722497    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:19:52.722609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:52.722543    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:19:53.036919 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:53.036883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" event={"ID":"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2","Type":"ContainerStarted","Data":"946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406"}
Sep 17 21:19:53.037156 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:53.036932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" event={"ID":"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2","Type":"ContainerStarted","Data":"4192aae4da68b0b2061d2aee9064af512d9ba415d0de44ad399b0cfeef245a07"}
Sep 17 21:19:53.063726 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:53.063667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" podStartSLOduration=1.063652044 podStartE2EDuration="1.063652044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:53.061199352 +0000 UTC m=+1831.880491315" watchObservedRunningTime="2026-09-17 21:19:53.063652044 +0000 UTC m=+1831.882944006"
Sep 17 21:19:53.441295 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:53.441189    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:53.446155 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:53.446129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:54.041996 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:54.041965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:54.042850 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:54.042825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:19:56.099213 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.099181    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"]
Sep 17 21:19:56.102412 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.102355    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.108263 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.108222    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-sk4c7\""
Sep 17 21:19:56.108386 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.108344    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-8pwgf\""
Sep 17 21:19:56.108386 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.108346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 21:19:56.108505 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.108413    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 21:19:56.108620 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.108604    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 21:19:56.116389 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.116366    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"]
Sep 17 21:19:56.180259 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.180210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.180435 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.180300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnt9\" (UniqueName: \"kubernetes.io/projected/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-kube-api-access-gdnt9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.281586 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.281546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnt9\" (UniqueName: \"kubernetes.io/projected/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-kube-api-access-gdnt9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.281779 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.281644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.284358 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.284331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.298522 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.298488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnt9\" (UniqueName: \"kubernetes.io/projected/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-kube-api-access-gdnt9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.413406 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.413320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:56.695329 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:56.695304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"]
Sep 17 21:19:56.696798 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:19:56.696769    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode251d5e4_4c70_4ee7_9eb6_41ff25e5ec7a.slice/crio-490ed4b77d715603e030c5cc3b21ea558e14a63b0df3776ff51b54fc4feddb2d WatchSource:0}: Error finding container 490ed4b77d715603e030c5cc3b21ea558e14a63b0df3776ff51b54fc4feddb2d: Status 404 returned error can't find the container with id 490ed4b77d715603e030c5cc3b21ea558e14a63b0df3776ff51b54fc4feddb2d
Sep 17 21:19:57.057410 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:57.057378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" event={"ID":"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a","Type":"ContainerStarted","Data":"8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf"}
Sep 17 21:19:57.057609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:57.057427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" event={"ID":"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a","Type":"ContainerStarted","Data":"490ed4b77d715603e030c5cc3b21ea558e14a63b0df3776ff51b54fc4feddb2d"}
Sep 17 21:19:57.057609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:57.057523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:19:57.083025 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:19:57.082972    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" podStartSLOduration=1.082951125 podStartE2EDuration="1.082951125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:57.080656871 +0000 UTC m=+1835.899948834" watchObservedRunningTime="2026-09-17 21:19:57.082951125 +0000 UTC m=+1835.902243087"
Sep 17 21:20:28.064042 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:28.064007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:20:46.312324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:46.312282    2577 generic.go:356] "Generic (PLEG): container finished" podID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerID="946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406" exitCode=0
Sep 17 21:20:46.312753 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:46.312342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" event={"ID":"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2","Type":"ContainerDied","Data":"946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406"}
Sep 17 21:20:46.312753 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:46.312702    2577 scope.go:117] "RemoveContainer" containerID="946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406"
Sep 17 21:20:46.313659 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:46.313626    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:20:46.313729 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:46.313713    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:20:46.313781 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:46.313765    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:20:47.319166 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:47.319131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" event={"ID":"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2","Type":"ContainerStarted","Data":"f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a"}
Sep 17 21:20:47.440726 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:47.440681    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:20:47.440993 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:47.440956    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 17 21:20:48.441015 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:48.440962    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 17 21:20:49.441478 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:49.441429    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 17 21:20:50.441223 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:50.441177    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 17 21:20:51.441033 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:51.440986    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 17 21:20:52.444538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:52.444505    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:20:52.444921 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:52.444790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:20:52.445518 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:52.445500    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:20:56.602625 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:56.602593    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"]
Sep 17 21:20:58.064420 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:20:58.064376    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:21:07.645618 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:07.645586    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"]
Sep 17 21:21:07.646079 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:07.645784    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy" containerID="cri-o://f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a" gracePeriod=30
Sep 17 21:21:08.063796 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:08.063757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:21:11.622024 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:11.621976    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main" containerID="cri-o://8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf" gracePeriod=45
Sep 17 21:21:12.889332 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:12.889303    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:21:13.010619 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istiod-ca-cert\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.010760 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-podinfo\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.010760 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010664    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-envoy\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.010902 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-certs\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.010902 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-data\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.010902 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010868    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-socket\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.011060 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-credential-socket\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.011060 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g54wp\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-kube-api-access-g54wp\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.011060 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:21:13.011228 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:21:13.011228 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011075    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:21:13.011228 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.010996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-token\") pod \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\" (UID: \"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2\") "
Sep 17 21:21:13.011228 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:21:13.011467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:21:13.011467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011422    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.011467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011440    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-data\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.011467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011453    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-workload-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.011467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011465    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-credential-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.011656 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.011479    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istiod-ca-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.013374 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.013334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-kube-api-access-g54wp" (OuterVolumeSpecName: "kube-api-access-g54wp") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "kube-api-access-g54wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:21:13.013487 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.013394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:21:13.013487 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.013428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:21:13.013487 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.013450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" (UID: "330bdcf1-42c4-4d07-9cbf-6ffc203b21c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 21:21:13.112829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.112800    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-podinfo\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.112829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.112825    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-envoy\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.112829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.112834    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54wp\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-kube-api-access-g54wp\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.112829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.112844    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2-istio-token\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:13.452673 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.452640    2577 generic.go:356] "Generic (PLEG): container finished" podID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerID="f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a" exitCode=0
Sep 17 21:21:13.452973 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.452689    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"
Sep 17 21:21:13.452973 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.452732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" event={"ID":"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2","Type":"ContainerDied","Data":"f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a"}
Sep 17 21:21:13.452973 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.452800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb" event={"ID":"330bdcf1-42c4-4d07-9cbf-6ffc203b21c2","Type":"ContainerDied","Data":"4192aae4da68b0b2061d2aee9064af512d9ba415d0de44ad399b0cfeef245a07"}
Sep 17 21:21:13.452973 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.452831    2577 scope.go:117] "RemoveContainer" containerID="f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a"
Sep 17 21:21:13.463506 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.463490    2577 scope.go:117] "RemoveContainer" containerID="946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406"
Sep 17 21:21:13.471423 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.471408    2577 scope.go:117] "RemoveContainer" containerID="f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a"
Sep 17 21:21:13.471646 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:21:13.471629    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a\": container with ID starting with f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a not found: ID does not exist" containerID="f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a"
Sep 17 21:21:13.471703 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.471652    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a"} err="failed to get container status \"f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a\": rpc error: code = NotFound desc = could not find container \"f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a\": container with ID starting with f777aab7da24e85833ba904e67d5e2e8b4a554487491cbe45b6d5dfde5b9ed6a not found: ID does not exist"
Sep 17 21:21:13.471703 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.471666    2577 scope.go:117] "RemoveContainer" containerID="946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406"
Sep 17 21:21:13.471904 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:21:13.471889    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406\": container with ID starting with 946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406 not found: ID does not exist" containerID="946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406"
Sep 17 21:21:13.471952 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.471906    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406"} err="failed to get container status \"946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406\": rpc error: code = NotFound desc = could not find container \"946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406\": container with ID starting with 946a041ebfbcac71fffeac297f646fcd78a7faeef73bd3b89cd95d0a55680406 not found: ID does not exist"
Sep 17 21:21:13.475671 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.475649    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"]
Sep 17 21:21:13.479281 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.479261    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-546nb"]
Sep 17 21:21:13.724269 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:13.724168    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" path="/var/lib/kubelet/pods/330bdcf1-42c4-4d07-9cbf-6ffc203b21c2/volumes"
Sep 17 21:21:18.063443 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:18.063401    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:21:18.063855 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:18.063516    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:21:21.584942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.584902    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"]
Sep 17 21:21:21.585480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.585376    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy"
Sep 17 21:21:21.585480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.585395    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy"
Sep 17 21:21:21.585480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.585413    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy"
Sep 17 21:21:21.585480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.585422    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy"
Sep 17 21:21:21.585695 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.585513    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy"
Sep 17 21:21:21.585695 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.585525    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="330bdcf1-42c4-4d07-9cbf-6ffc203b21c2" containerName="istio-proxy"
Sep 17 21:21:21.590143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.590119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:21.593038 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.593006    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 21:21:21.593165 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.593082    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 21:21:21.595591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.595392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 21:21:21.595591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.595448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-jqrgx\""
Sep 17 21:21:21.595591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.595571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-t7zhm\""
Sep 17 21:21:21.602419 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.601675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"]
Sep 17 21:21:21.691479 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.691443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkbc\" (UniqueName: \"kubernetes.io/projected/a90222b8-0bd9-4336-99ac-7083b6bd752e-kube-api-access-lfkbc\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:21.691659 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.691502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90222b8-0bd9-4336-99ac-7083b6bd752e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:21.792057 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.792019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkbc\" (UniqueName: \"kubernetes.io/projected/a90222b8-0bd9-4336-99ac-7083b6bd752e-kube-api-access-lfkbc\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:21.792057 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.792063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90222b8-0bd9-4336-99ac-7083b6bd752e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:21.794637 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.794613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90222b8-0bd9-4336-99ac-7083b6bd752e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:21.801454 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.801430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkbc\" (UniqueName: \"kubernetes.io/projected/a90222b8-0bd9-4336-99ac-7083b6bd752e-kube-api-access-lfkbc\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:21.906080 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:21.905998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:22.184004 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:22.183979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"]
Sep 17 21:21:22.186208 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:21:22.186179    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90222b8_0bd9_4336_99ac_7083b6bd752e.slice/crio-5c47362626b046dc73b98be108054bd928a3776e22c34d598b1fb83f9a5364f8 WatchSource:0}: Error finding container 5c47362626b046dc73b98be108054bd928a3776e22c34d598b1fb83f9a5364f8: Status 404 returned error can't find the container with id 5c47362626b046dc73b98be108054bd928a3776e22c34d598b1fb83f9a5364f8
Sep 17 21:21:22.497644 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:22.497556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" event={"ID":"a90222b8-0bd9-4336-99ac-7083b6bd752e","Type":"ContainerStarted","Data":"cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4"}
Sep 17 21:21:22.497644 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:22.497618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" event={"ID":"a90222b8-0bd9-4336-99ac-7083b6bd752e","Type":"ContainerStarted","Data":"5c47362626b046dc73b98be108054bd928a3776e22c34d598b1fb83f9a5364f8"}
Sep 17 21:21:22.497874 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:22.497654    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:21:22.518669 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:22.518595    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" podStartSLOduration=1.518578166 podStartE2EDuration="1.518578166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:22.51731248 +0000 UTC m=+1921.336604443" watchObservedRunningTime="2026-09-17 21:21:22.518578166 +0000 UTC m=+1921.337870127"
Sep 17 21:21:28.063867 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:28.063817    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:21:38.064070 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:38.064029    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:21:41.872008 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:41.871980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:21:41.940997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:41.940915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-tls-certs\") pod \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") "
Sep 17 21:21:41.940997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:41.940996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdnt9\" (UniqueName: \"kubernetes.io/projected/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-kube-api-access-gdnt9\") pod \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\" (UID: \"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a\") "
Sep 17 21:21:41.943342 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:41.943307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" (UID: "e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:21:41.943461 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:41.943342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-kube-api-access-gdnt9" (OuterVolumeSpecName: "kube-api-access-gdnt9") pod "e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" (UID: "e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a"). InnerVolumeSpecName "kube-api-access-gdnt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:21:42.041614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.041576    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:42.041614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.041615    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdnt9\" (UniqueName: \"kubernetes.io/projected/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a-kube-api-access-gdnt9\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:21:42.591913 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.591880    2577 generic.go:356] "Generic (PLEG): container finished" podID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerID="8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf" exitCode=0
Sep 17 21:21:42.592116 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.591923    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"
Sep 17 21:21:42.592116 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.591924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" event={"ID":"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a","Type":"ContainerDied","Data":"8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf"}
Sep 17 21:21:42.592116 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.592057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl" event={"ID":"e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a","Type":"ContainerDied","Data":"490ed4b77d715603e030c5cc3b21ea558e14a63b0df3776ff51b54fc4feddb2d"}
Sep 17 21:21:42.592116 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.592092    2577 scope.go:117] "RemoveContainer" containerID="8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf"
Sep 17 21:21:42.601083 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.601062    2577 scope.go:117] "RemoveContainer" containerID="8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf"
Sep 17 21:21:42.601348 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:21:42.601328    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf\": container with ID starting with 8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf not found: ID does not exist" containerID="8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf"
Sep 17 21:21:42.601416 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.601356    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf"} err="failed to get container status \"8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf\": rpc error: code = NotFound desc = could not find container \"8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf\": container with ID starting with 8be08a27a1fffa8043fddb9a84da5d3abf67838fa3542ae2f5fa91d07a784aaf not found: ID does not exist"
Sep 17 21:21:42.617763 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.617741    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"]
Sep 17 21:21:42.619521 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:42.619502    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4hlrl"]
Sep 17 21:21:43.717683 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:43.717648    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" path="/var/lib/kubelet/pods/e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a/volumes"
Sep 17 21:21:53.505228 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:21:53.505197    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:22:17.866428 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.866349    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"]
Sep 17 21:22:17.866806 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.866678    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main"
Sep 17 21:22:17.866806 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.866690    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main"
Sep 17 21:22:17.866806 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.866755    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e251d5e4-4c70-4ee7-9eb6-41ff25e5ec7a" containerName="main"
Sep 17 21:22:17.869455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.869433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:17.875762 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.875743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 21:22:17.876000 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.875975    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 21:22:17.876121 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.876001    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-x7x7h\""
Sep 17 21:22:17.876220 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.876193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-6mjf7\""
Sep 17 21:22:17.876340 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.876293    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 21:22:17.896832 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.896808    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"]
Sep 17 21:22:17.921809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.921781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7svg\" (UniqueName: \"kubernetes.io/projected/23adec31-478d-4ef9-a71b-97ce6cf4be85-kube-api-access-k7svg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:17.921925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:17.921843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23adec31-478d-4ef9-a71b-97ce6cf4be85-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:18.023327 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.023294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23adec31-478d-4ef9-a71b-97ce6cf4be85-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:18.023518 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.023358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7svg\" (UniqueName: \"kubernetes.io/projected/23adec31-478d-4ef9-a71b-97ce6cf4be85-kube-api-access-k7svg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:18.025928 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.025903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23adec31-478d-4ef9-a71b-97ce6cf4be85-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:18.033834 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.033810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7svg\" (UniqueName: \"kubernetes.io/projected/23adec31-478d-4ef9-a71b-97ce6cf4be85-kube-api-access-k7svg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:18.179474 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.179386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:18.467160 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.467136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"]
Sep 17 21:22:18.469914 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:22:18.469880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23adec31_478d_4ef9_a71b_97ce6cf4be85.slice/crio-91ac48fa5fabe421f72067eb313ed04c101f96608e465c1fec5627c1b9e53c98 WatchSource:0}: Error finding container 91ac48fa5fabe421f72067eb313ed04c101f96608e465c1fec5627c1b9e53c98: Status 404 returned error can't find the container with id 91ac48fa5fabe421f72067eb313ed04c101f96608e465c1fec5627c1b9e53c98
Sep 17 21:22:18.760729 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.760692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" event={"ID":"23adec31-478d-4ef9-a71b-97ce6cf4be85","Type":"ContainerStarted","Data":"8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c"}
Sep 17 21:22:18.760887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.760751    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:18.760887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:18.760768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" event={"ID":"23adec31-478d-4ef9-a71b-97ce6cf4be85","Type":"ContainerStarted","Data":"91ac48fa5fabe421f72067eb313ed04c101f96608e465c1fec5627c1b9e53c98"}
Sep 17 21:22:49.768393 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:49.768361    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:22:49.799159 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:22:49.799101    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" podStartSLOduration=32.799081924 podStartE2EDuration="32.799081924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:22:18.78293832 +0000 UTC m=+1977.602230295" watchObservedRunningTime="2026-09-17 21:22:49.799081924 +0000 UTC m=+2008.618373885"
Sep 17 21:23:11.802415 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.802379    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"]
Sep 17 21:23:11.805821 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.805798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.808206 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.808178    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 21:23:11.816113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.816084    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"]
Sep 17 21:23:11.878663 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.878629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.878663 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.878673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.878887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.878724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.878887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.878747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.878887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.878765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.878887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.878794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vtz\" (UniqueName: \"kubernetes.io/projected/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-kube-api-access-r4vtz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980084 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980084 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980441 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980441 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vtz\" (UniqueName: \"kubernetes.io/projected/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-kube-api-access-r4vtz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980539 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980596 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.980596 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.980585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.982784 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.982754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.983065 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.983042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:11.993972 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:11.993942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vtz\" (UniqueName: \"kubernetes.io/projected/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-kube-api-access-r4vtz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:12.118268 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:12.118158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:12.393605 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:12.393578    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"]
Sep 17 21:23:12.395339 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:23:12.395307    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa2b4e5_fbf7_44ca_9021_cd324a2351b3.slice/crio-6428709103d90382edccdf490f8d7d3ea4d356720b2c052295eaf71bc9bccd4a WatchSource:0}: Error finding container 6428709103d90382edccdf490f8d7d3ea4d356720b2c052295eaf71bc9bccd4a: Status 404 returned error can't find the container with id 6428709103d90382edccdf490f8d7d3ea4d356720b2c052295eaf71bc9bccd4a
Sep 17 21:23:13.026204 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:13.026168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" event={"ID":"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3","Type":"ContainerStarted","Data":"417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965"}
Sep 17 21:23:13.026674 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:13.026222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" event={"ID":"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3","Type":"ContainerStarted","Data":"6428709103d90382edccdf490f8d7d3ea4d356720b2c052295eaf71bc9bccd4a"}
Sep 17 21:23:13.044560 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:13.044503    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" podStartSLOduration=2.044486857 podStartE2EDuration="2.044486857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:23:13.044062948 +0000 UTC m=+2031.863354923" watchObservedRunningTime="2026-09-17 21:23:13.044486857 +0000 UTC m=+2031.863778818"
Sep 17 21:23:13.118677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:13.118643    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:22.118337 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:22.118293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:22.130769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:22.130742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:23.087579 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:23.087549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:45.247911 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.247873    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"]
Sep 17 21:23:45.257564 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:23:45.257514    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:23:45.257564 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:23:45.257564 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:23:45.257564 ip-10-0-134-35 kubenswrapper[2577]:  > containerID="417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965" cmd=["/bin/sleep","15"]
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.257590    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" message="" err=<
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]:  >
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:23:45.257607    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" podUID="cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" containerName="main" containerID="cri-o://417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965"
Sep 17 21:23:45.257800 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.257700    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" podUID="cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" containerName="main" containerID="cri-o://417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965" gracePeriod=60
Sep 17 21:23:45.604160 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.604135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:45.763123 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-home\") pod \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") "
Sep 17 21:23:45.763123 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763125    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-dshm\") pod \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") "
Sep 17 21:23:45.763383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763151    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tmp-dir\") pod \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") "
Sep 17 21:23:45.763383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-model-cache\") pod \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") "
Sep 17 21:23:45.763383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763294    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tls-certs\") pod \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") "
Sep 17 21:23:45.763383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4vtz\" (UniqueName: \"kubernetes.io/projected/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-kube-api-access-r4vtz\") pod \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\" (UID: \"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3\") "
Sep 17 21:23:45.763609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763424    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-home" (OuterVolumeSpecName: "home") pod "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" (UID: "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:23:45.763609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" (UID: "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:23:45.763609 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" (UID: "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:23:45.763780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763631    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:23:45.763780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.763647    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:23:45.765594 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.765570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-kube-api-access-r4vtz" (OuterVolumeSpecName: "kube-api-access-r4vtz") pod "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" (UID: "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3"). InnerVolumeSpecName "kube-api-access-r4vtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:23:45.765594 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.765575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-dshm" (OuterVolumeSpecName: "dshm") pod "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" (UID: "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:23:45.765739 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.765654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" (UID: "cfa2b4e5-fbf7-44ca-9021-cd324a2351b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:23:45.864455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.864416    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:23:45.864455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.864447    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4vtz\" (UniqueName: \"kubernetes.io/projected/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-kube-api-access-r4vtz\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:23:45.864455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.864457    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:23:45.864455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:45.864465    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:23:46.190108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.190020    2577 generic.go:356] "Generic (PLEG): container finished" podID="cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" containerID="417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965" exitCode=0
Sep 17 21:23:46.190108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.190061    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"
Sep 17 21:23:46.190392 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.190112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" event={"ID":"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3","Type":"ContainerDied","Data":"417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965"}
Sep 17 21:23:46.190392 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.190171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w" event={"ID":"cfa2b4e5-fbf7-44ca-9021-cd324a2351b3","Type":"ContainerDied","Data":"6428709103d90382edccdf490f8d7d3ea4d356720b2c052295eaf71bc9bccd4a"}
Sep 17 21:23:46.190392 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.190199    2577 scope.go:117] "RemoveContainer" containerID="417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965"
Sep 17 21:23:46.200157 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.200141    2577 scope.go:117] "RemoveContainer" containerID="417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965"
Sep 17 21:23:46.200457 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:23:46.200438    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965\": container with ID starting with 417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965 not found: ID does not exist" containerID="417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965"
Sep 17 21:23:46.200525 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.200468    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965"} err="failed to get container status \"417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965\": rpc error: code = NotFound desc = could not find container \"417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965\": container with ID starting with 417fe9872b89d2986181591a96064e668ed19a7e8027a407652cfe0994833965 not found: ID does not exist"
Sep 17 21:23:46.223173 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.223139    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"]
Sep 17 21:23:46.231836 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:46.231809    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995pxz2w"]
Sep 17 21:23:47.717809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:47.717772    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" path="/var/lib/kubelet/pods/cfa2b4e5-fbf7-44ca-9021-cd324a2351b3/volumes"
Sep 17 21:23:48.031892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:48.031857    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"]
Sep 17 21:23:53.504959 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:23:53.504924    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:03.052020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:03.051967    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main" containerID="cri-o://cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4" gracePeriod=45
Sep 17 21:24:03.504890 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:03.504799    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:13.504641 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:13.504596    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:13.505129 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:13.504734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:24:23.505429 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:23.505392    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:33.301776 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.301750    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:24:33.416349 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.416264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfkbc\" (UniqueName: \"kubernetes.io/projected/a90222b8-0bd9-4336-99ac-7083b6bd752e-kube-api-access-lfkbc\") pod \"a90222b8-0bd9-4336-99ac-7083b6bd752e\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") "
Sep 17 21:24:33.416349 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.416325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90222b8-0bd9-4336-99ac-7083b6bd752e-tls-certs\") pod \"a90222b8-0bd9-4336-99ac-7083b6bd752e\" (UID: \"a90222b8-0bd9-4336-99ac-7083b6bd752e\") "
Sep 17 21:24:33.418765 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.418743    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90222b8-0bd9-4336-99ac-7083b6bd752e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a90222b8-0bd9-4336-99ac-7083b6bd752e" (UID: "a90222b8-0bd9-4336-99ac-7083b6bd752e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:24:33.418895 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.418863    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90222b8-0bd9-4336-99ac-7083b6bd752e-kube-api-access-lfkbc" (OuterVolumeSpecName: "kube-api-access-lfkbc") pod "a90222b8-0bd9-4336-99ac-7083b6bd752e" (UID: "a90222b8-0bd9-4336-99ac-7083b6bd752e"). InnerVolumeSpecName "kube-api-access-lfkbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:24:33.422466 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.422441    2577 generic.go:356] "Generic (PLEG): container finished" podID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerID="cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4" exitCode=0
Sep 17 21:24:33.422568 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.422488    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"
Sep 17 21:24:33.422568 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.422524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" event={"ID":"a90222b8-0bd9-4336-99ac-7083b6bd752e","Type":"ContainerDied","Data":"cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4"}
Sep 17 21:24:33.422659 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.422578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs" event={"ID":"a90222b8-0bd9-4336-99ac-7083b6bd752e","Type":"ContainerDied","Data":"5c47362626b046dc73b98be108054bd928a3776e22c34d598b1fb83f9a5364f8"}
Sep 17 21:24:33.422659 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.422600    2577 scope.go:117] "RemoveContainer" containerID="cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4"
Sep 17 21:24:33.434140 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.434123    2577 scope.go:117] "RemoveContainer" containerID="cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4"
Sep 17 21:24:33.434429 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:24:33.434410    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4\": container with ID starting with cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4 not found: ID does not exist" containerID="cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4"
Sep 17 21:24:33.434502 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.434440    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4"} err="failed to get container status \"cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4\": rpc error: code = NotFound desc = could not find container \"cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4\": container with ID starting with cbbe690137f33de242a46f9c8f7dcfebe34fb283e57495fee5e341653c8439a4 not found: ID does not exist"
Sep 17 21:24:33.445434 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.445403    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"]
Sep 17 21:24:33.451205 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.451174    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesg7vs"]
Sep 17 21:24:33.516830 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.516801    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90222b8-0bd9-4336-99ac-7083b6bd752e-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:24:33.516830 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.516830    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfkbc\" (UniqueName: \"kubernetes.io/projected/a90222b8-0bd9-4336-99ac-7083b6bd752e-kube-api-access-lfkbc\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:24:33.717225 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:33.717131    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" path="/var/lib/kubelet/pods/a90222b8-0bd9-4336-99ac-7083b6bd752e/volumes"
Sep 17 21:24:35.235105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:35.235067    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"]
Sep 17 21:24:39.768449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:39.768412    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:48.699994 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.699949    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"]
Sep 17 21:24:48.700546 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.700475    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main"
Sep 17 21:24:48.700546 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.700495    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main"
Sep 17 21:24:48.700546 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.700512    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" containerName="main"
Sep 17 21:24:48.700546 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.700520    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" containerName="main"
Sep 17 21:24:48.700805 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.700610    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa2b4e5-fbf7-44ca-9021-cd324a2351b3" containerName="main"
Sep 17 21:24:48.700805 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.700623    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90222b8-0bd9-4336-99ac-7083b6bd752e" containerName="main"
Sep 17 21:24:48.705066 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.705043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:48.708443 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.708418    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 21:24:48.708620 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.708430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 21:24:48.708725 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.708432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-gqrrx\""
Sep 17 21:24:48.708725 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.708547    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 21:24:48.708851 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.708561    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-6tf4r\""
Sep 17 21:24:48.721754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.721720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"]
Sep 17 21:24:48.838360 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.838325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140d9079-f96d-4fa0-974b-70538bcff2bb-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:48.838512 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.838368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dsq\" (UniqueName: \"kubernetes.io/projected/140d9079-f96d-4fa0-974b-70538bcff2bb-kube-api-access-n4dsq\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:48.938766 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.938736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140d9079-f96d-4fa0-974b-70538bcff2bb-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:48.938947 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.938776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dsq\" (UniqueName: \"kubernetes.io/projected/140d9079-f96d-4fa0-974b-70538bcff2bb-kube-api-access-n4dsq\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:48.941395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.941369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140d9079-f96d-4fa0-974b-70538bcff2bb-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:48.947105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:48.947081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dsq\" (UniqueName: \"kubernetes.io/projected/140d9079-f96d-4fa0-974b-70538bcff2bb-kube-api-access-n4dsq\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:49.018178 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:49.018148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:49.294664 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:49.294635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"]
Sep 17 21:24:49.296381 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:24:49.296353    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140d9079_f96d_4fa0_974b_70538bcff2bb.slice/crio-abea3ee513d2e3fb15ab1b247b034445cde02ece8b85164589acff2e0d4b7997 WatchSource:0}: Error finding container abea3ee513d2e3fb15ab1b247b034445cde02ece8b85164589acff2e0d4b7997: Status 404 returned error can't find the container with id abea3ee513d2e3fb15ab1b247b034445cde02ece8b85164589acff2e0d4b7997
Sep 17 21:24:49.500342 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:49.500309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" event={"ID":"140d9079-f96d-4fa0-974b-70538bcff2bb","Type":"ContainerStarted","Data":"1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349"}
Sep 17 21:24:49.500538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:49.500359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" event={"ID":"140d9079-f96d-4fa0-974b-70538bcff2bb","Type":"ContainerStarted","Data":"abea3ee513d2e3fb15ab1b247b034445cde02ece8b85164589acff2e0d4b7997"}
Sep 17 21:24:49.500538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:49.500439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:24:49.520653 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:49.520609    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" podStartSLOduration=1.520597055 podStartE2EDuration="1.520597055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:24:49.518430561 +0000 UTC m=+2128.337722526" watchObservedRunningTime="2026-09-17 21:24:49.520597055 +0000 UTC m=+2128.339889017"
Sep 17 21:24:49.767759 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:49.767718    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:50.254952 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:50.254837    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main" containerID="cri-o://8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c" gracePeriod=45
Sep 17 21:24:59.767596 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:59.767554    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:59.768073 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:24:59.767659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:25:09.768110 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:09.768068    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:25:19.768233 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:19.768136    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:25:20.499782 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.499749    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:25:20.506509 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.506488    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:25:20.595347 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.595269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23adec31-478d-4ef9-a71b-97ce6cf4be85-tls-certs\") pod \"23adec31-478d-4ef9-a71b-97ce6cf4be85\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") "
Sep 17 21:25:20.595347 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.595338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7svg\" (UniqueName: \"kubernetes.io/projected/23adec31-478d-4ef9-a71b-97ce6cf4be85-kube-api-access-k7svg\") pod \"23adec31-478d-4ef9-a71b-97ce6cf4be85\" (UID: \"23adec31-478d-4ef9-a71b-97ce6cf4be85\") "
Sep 17 21:25:20.597547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.597518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23adec31-478d-4ef9-a71b-97ce6cf4be85-kube-api-access-k7svg" (OuterVolumeSpecName: "kube-api-access-k7svg") pod "23adec31-478d-4ef9-a71b-97ce6cf4be85" (UID: "23adec31-478d-4ef9-a71b-97ce6cf4be85"). InnerVolumeSpecName "kube-api-access-k7svg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:25:20.597665 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.597561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23adec31-478d-4ef9-a71b-97ce6cf4be85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23adec31-478d-4ef9-a71b-97ce6cf4be85" (UID: "23adec31-478d-4ef9-a71b-97ce6cf4be85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:25:20.654514 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.654479    2577 generic.go:356] "Generic (PLEG): container finished" podID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerID="8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c" exitCode=0
Sep 17 21:25:20.654650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.654523    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"
Sep 17 21:25:20.654650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.654567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" event={"ID":"23adec31-478d-4ef9-a71b-97ce6cf4be85","Type":"ContainerDied","Data":"8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c"}
Sep 17 21:25:20.654650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.654615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68" event={"ID":"23adec31-478d-4ef9-a71b-97ce6cf4be85","Type":"ContainerDied","Data":"91ac48fa5fabe421f72067eb313ed04c101f96608e465c1fec5627c1b9e53c98"}
Sep 17 21:25:20.654650 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.654637    2577 scope.go:117] "RemoveContainer" containerID="8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c"
Sep 17 21:25:20.663511 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.663496    2577 scope.go:117] "RemoveContainer" containerID="8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c"
Sep 17 21:25:20.663800 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:25:20.663775    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c\": container with ID starting with 8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c not found: ID does not exist" containerID="8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c"
Sep 17 21:25:20.663903 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.663803    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c"} err="failed to get container status \"8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c\": rpc error: code = NotFound desc = could not find container \"8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c\": container with ID starting with 8026a106e3cdfb307153d406810e74febb6738a16154a137a3734dcb0f5de31c not found: ID does not exist"
Sep 17 21:25:20.675837 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.675809    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"]
Sep 17 21:25:20.677618 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.677600    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schelsk68"]
Sep 17 21:25:20.696541 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.696515    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7svg\" (UniqueName: \"kubernetes.io/projected/23adec31-478d-4ef9-a71b-97ce6cf4be85-kube-api-access-k7svg\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:25:20.696541 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:20.696540    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23adec31-478d-4ef9-a71b-97ce6cf4be85-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:25:21.717359 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:21.717325    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" path="/var/lib/kubelet/pods/23adec31-478d-4ef9-a71b-97ce6cf4be85/volumes"
Sep 17 21:25:32.320948 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.320910    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"]
Sep 17 21:25:32.321499 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.321335    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main"
Sep 17 21:25:32.321499 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.321353    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main"
Sep 17 21:25:32.321499 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.321406    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23adec31-478d-4ef9-a71b-97ce6cf4be85" containerName="main"
Sep 17 21:25:32.324287 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.324264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.330072 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.330045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 21:25:32.330199 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.330045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-jbgp5\""
Sep 17 21:25:32.330199 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.330092    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-5pmv8\""
Sep 17 21:25:32.330527 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.330506    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 21:25:32.330640 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.330543    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 21:25:32.339274 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.339216    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"]
Sep 17 21:25:32.387395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.387363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f001ae9d-9017-4d32-88da-6ec48254eb58-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.387528 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.387415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9qw\" (UniqueName: \"kubernetes.io/projected/f001ae9d-9017-4d32-88da-6ec48254eb58-kube-api-access-gb9qw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.488453 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.488419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9qw\" (UniqueName: \"kubernetes.io/projected/f001ae9d-9017-4d32-88da-6ec48254eb58-kube-api-access-gb9qw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.488616 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.488508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f001ae9d-9017-4d32-88da-6ec48254eb58-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.491113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.491090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f001ae9d-9017-4d32-88da-6ec48254eb58-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.496750 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.496726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9qw\" (UniqueName: \"kubernetes.io/projected/f001ae9d-9017-4d32-88da-6ec48254eb58-kube-api-access-gb9qw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.634795 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.634705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:32.908061 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:32.908027    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"]
Sep 17 21:25:32.910315 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:25:32.910289    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf001ae9d_9017_4d32_88da_6ec48254eb58.slice/crio-4f74b0fe0f2cba468b4c27b3b74bc4b278c8c919029b05f7f25c15cdfc476710 WatchSource:0}: Error finding container 4f74b0fe0f2cba468b4c27b3b74bc4b278c8c919029b05f7f25c15cdfc476710: Status 404 returned error can't find the container with id 4f74b0fe0f2cba468b4c27b3b74bc4b278c8c919029b05f7f25c15cdfc476710
Sep 17 21:25:33.720225 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:33.720184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" event={"ID":"f001ae9d-9017-4d32-88da-6ec48254eb58","Type":"ContainerStarted","Data":"3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474"}
Sep 17 21:25:33.720625 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:33.720259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" event={"ID":"f001ae9d-9017-4d32-88da-6ec48254eb58","Type":"ContainerStarted","Data":"4f74b0fe0f2cba468b4c27b3b74bc4b278c8c919029b05f7f25c15cdfc476710"}
Sep 17 21:25:33.720625 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:33.720406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:25:33.738821 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:25:33.738779    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" podStartSLOduration=1.7387657829999998 podStartE2EDuration="1.738765783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:25:33.73763997 +0000 UTC m=+2172.556931933" watchObservedRunningTime="2026-09-17 21:25:33.738765783 +0000 UTC m=+2172.558057745"
Sep 17 21:26:04.726721 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:26:04.726689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:26:54.270001 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:26:54.269967    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"]
Sep 17 21:27:00.507458 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:00.507415    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:09.290259 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:09.290192    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main" containerID="cri-o://1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349" gracePeriod=45
Sep 17 21:27:10.507041 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:10.507000    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:20.507325 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:20.507283    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:20.507791 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:20.507419    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:27:30.507037 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:30.506995    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:39.539701 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:39.539675    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:27:39.655947 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:39.655854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4dsq\" (UniqueName: \"kubernetes.io/projected/140d9079-f96d-4fa0-974b-70538bcff2bb-kube-api-access-n4dsq\") pod \"140d9079-f96d-4fa0-974b-70538bcff2bb\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") "
Sep 17 21:27:39.655947 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:39.655917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140d9079-f96d-4fa0-974b-70538bcff2bb-tls-certs\") pod \"140d9079-f96d-4fa0-974b-70538bcff2bb\" (UID: \"140d9079-f96d-4fa0-974b-70538bcff2bb\") "
Sep 17 21:27:39.658222 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:39.658191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140d9079-f96d-4fa0-974b-70538bcff2bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "140d9079-f96d-4fa0-974b-70538bcff2bb" (UID: "140d9079-f96d-4fa0-974b-70538bcff2bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:27:39.658358 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:39.658232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140d9079-f96d-4fa0-974b-70538bcff2bb-kube-api-access-n4dsq" (OuterVolumeSpecName: "kube-api-access-n4dsq") pod "140d9079-f96d-4fa0-974b-70538bcff2bb" (UID: "140d9079-f96d-4fa0-974b-70538bcff2bb"). InnerVolumeSpecName "kube-api-access-n4dsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:27:39.758270 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:39.757573    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dsq\" (UniqueName: \"kubernetes.io/projected/140d9079-f96d-4fa0-974b-70538bcff2bb-kube-api-access-n4dsq\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:27:39.758270 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:39.757621    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/140d9079-f96d-4fa0-974b-70538bcff2bb-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:27:40.323904 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.323865    2577 generic.go:356] "Generic (PLEG): container finished" podID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerID="1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349" exitCode=0
Sep 17 21:27:40.324107 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.323928    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"
Sep 17 21:27:40.324107 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.323963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" event={"ID":"140d9079-f96d-4fa0-974b-70538bcff2bb","Type":"ContainerDied","Data":"1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349"}
Sep 17 21:27:40.324107 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.324035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx" event={"ID":"140d9079-f96d-4fa0-974b-70538bcff2bb","Type":"ContainerDied","Data":"abea3ee513d2e3fb15ab1b247b034445cde02ece8b85164589acff2e0d4b7997"}
Sep 17 21:27:40.324107 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.324057    2577 scope.go:117] "RemoveContainer" containerID="1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349"
Sep 17 21:27:40.332528 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.332512    2577 scope.go:117] "RemoveContainer" containerID="1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349"
Sep 17 21:27:40.332791 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:27:40.332769    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349\": container with ID starting with 1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349 not found: ID does not exist" containerID="1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349"
Sep 17 21:27:40.332845 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.332799    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349"} err="failed to get container status \"1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349\": rpc error: code = NotFound desc = could not find container \"1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349\": container with ID starting with 1a1cac47e4ab6888125ba669cdefe77eb66d91f14b96a6880e104fc8c2980349 not found: ID does not exist"
Sep 17 21:27:40.344090 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.344067    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"]
Sep 17 21:27:40.347533 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:40.347513    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schesfhkx"]
Sep 17 21:27:41.718931 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:41.718900    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" path="/var/lib/kubelet/pods/140d9079-f96d-4fa0-974b-70538bcff2bb/volumes"
Sep 17 21:27:44.473791 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:44.473747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"]
Sep 17 21:27:44.727061 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:44.726983    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:52.501752 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.501718    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"]
Sep 17 21:27:52.502127 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.502068    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main"
Sep 17 21:27:52.502127 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.502080    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main"
Sep 17 21:27:52.502199 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.502137    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="140d9079-f96d-4fa0-974b-70538bcff2bb" containerName="main"
Sep 17 21:27:52.506415 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.506396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:52.510325 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.510298    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 21:27:52.510325 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.510298    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 21:27:52.510512 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.510327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-8xl82\""
Sep 17 21:27:52.510512 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.510351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 21:27:52.510512 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.510328    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-95c77\""
Sep 17 21:27:52.520415 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.520394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"]
Sep 17 21:27:52.665564 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.665525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6kbf\" (UniqueName: \"kubernetes.io/projected/15fca102-3e69-43ab-a17a-8d4cfd22e90a-kube-api-access-n6kbf\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:52.665751 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.665583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fca102-3e69-43ab-a17a-8d4cfd22e90a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:52.765976 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.765947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fca102-3e69-43ab-a17a-8d4cfd22e90a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:52.766150 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.766034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6kbf\" (UniqueName: \"kubernetes.io/projected/15fca102-3e69-43ab-a17a-8d4cfd22e90a-kube-api-access-n6kbf\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:52.768711 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.768685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fca102-3e69-43ab-a17a-8d4cfd22e90a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:52.774724 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.774705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6kbf\" (UniqueName: \"kubernetes.io/projected/15fca102-3e69-43ab-a17a-8d4cfd22e90a-kube-api-access-n6kbf\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:52.816607 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:52.816583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:53.093022 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:53.092993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"]
Sep 17 21:27:53.095000 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:27:53.094962    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fca102_3e69_43ab_a17a_8d4cfd22e90a.slice/crio-7df603200cd7b75e8bbcef078a3569fccb2c22306f83871c241561bd4f0e426e WatchSource:0}: Error finding container 7df603200cd7b75e8bbcef078a3569fccb2c22306f83871c241561bd4f0e426e: Status 404 returned error can't find the container with id 7df603200cd7b75e8bbcef078a3569fccb2c22306f83871c241561bd4f0e426e
Sep 17 21:27:53.389892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:53.389792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" event={"ID":"15fca102-3e69-43ab-a17a-8d4cfd22e90a","Type":"ContainerStarted","Data":"da40fc9ad9fe834a43f4fde84c529b2fc5ecdf1f0dbd19b0c61584ffbf26f3c1"}
Sep 17 21:27:53.389892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:53.389854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" event={"ID":"15fca102-3e69-43ab-a17a-8d4cfd22e90a","Type":"ContainerStarted","Data":"7df603200cd7b75e8bbcef078a3569fccb2c22306f83871c241561bd4f0e426e"}
Sep 17 21:27:53.389892 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:53.389880    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:27:53.411710 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:53.411658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" podStartSLOduration=1.411642601 podStartE2EDuration="1.411642601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:27:53.408878824 +0000 UTC m=+2312.228170787" watchObservedRunningTime="2026-09-17 21:27:53.411642601 +0000 UTC m=+2312.230934563"
Sep 17 21:27:54.727837 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:54.727794    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:59.493511 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:27:59.493443    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main" containerID="cri-o://3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474" gracePeriod=45
Sep 17 21:28:04.726529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:04.726484    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:28:04.727041 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:04.726587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:28:14.726453 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:14.726359    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:28:24.397470 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:24.397434    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:28:24.726223 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:24.726141    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:28:29.739024 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:29.738999    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:28:29.763660 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:29.763624    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb9qw\" (UniqueName: \"kubernetes.io/projected/f001ae9d-9017-4d32-88da-6ec48254eb58-kube-api-access-gb9qw\") pod \"f001ae9d-9017-4d32-88da-6ec48254eb58\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") "
Sep 17 21:28:29.763789 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:29.763732    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f001ae9d-9017-4d32-88da-6ec48254eb58-tls-certs\") pod \"f001ae9d-9017-4d32-88da-6ec48254eb58\" (UID: \"f001ae9d-9017-4d32-88da-6ec48254eb58\") "
Sep 17 21:28:29.766137 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:29.766101    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f001ae9d-9017-4d32-88da-6ec48254eb58-kube-api-access-gb9qw" (OuterVolumeSpecName: "kube-api-access-gb9qw") pod "f001ae9d-9017-4d32-88da-6ec48254eb58" (UID: "f001ae9d-9017-4d32-88da-6ec48254eb58"). InnerVolumeSpecName "kube-api-access-gb9qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:28:29.766137 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:29.766121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f001ae9d-9017-4d32-88da-6ec48254eb58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f001ae9d-9017-4d32-88da-6ec48254eb58" (UID: "f001ae9d-9017-4d32-88da-6ec48254eb58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:28:29.864407 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:29.864366    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9qw\" (UniqueName: \"kubernetes.io/projected/f001ae9d-9017-4d32-88da-6ec48254eb58-kube-api-access-gb9qw\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:28:29.864407 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:29.864399    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f001ae9d-9017-4d32-88da-6ec48254eb58-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:28:30.570503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.570459    2577 generic.go:356] "Generic (PLEG): container finished" podID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerID="3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474" exitCode=0
Sep 17 21:28:30.570713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.570518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"
Sep 17 21:28:30.570713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.570556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" event={"ID":"f001ae9d-9017-4d32-88da-6ec48254eb58","Type":"ContainerDied","Data":"3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474"}
Sep 17 21:28:30.570713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.570616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49" event={"ID":"f001ae9d-9017-4d32-88da-6ec48254eb58","Type":"ContainerDied","Data":"4f74b0fe0f2cba468b4c27b3b74bc4b278c8c919029b05f7f25c15cdfc476710"}
Sep 17 21:28:30.570713 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.570640    2577 scope.go:117] "RemoveContainer" containerID="3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474"
Sep 17 21:28:30.580114 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.580098    2577 scope.go:117] "RemoveContainer" containerID="3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474"
Sep 17 21:28:30.580383 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:28:30.580367    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474\": container with ID starting with 3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474 not found: ID does not exist" containerID="3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474"
Sep 17 21:28:30.580443 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.580390    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474"} err="failed to get container status \"3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474\": rpc error: code = NotFound desc = could not find container \"3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474\": container with ID starting with 3289d561532ff74e607e22fcef331a132319e18a42db668405d5cdc834536474 not found: ID does not exist"
Sep 17 21:28:30.595520 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.595492    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"]
Sep 17 21:28:30.597810 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:30.597787    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched7r49"]
Sep 17 21:28:31.717080 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:31.717044    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" path="/var/lib/kubelet/pods/f001ae9d-9017-4d32-88da-6ec48254eb58/volumes"
Sep 17 21:28:44.568490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.568458    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"]
Sep 17 21:28:44.568957 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.568758    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main"
Sep 17 21:28:44.568957 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.568768    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main"
Sep 17 21:28:44.568957 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.568827    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f001ae9d-9017-4d32-88da-6ec48254eb58" containerName="main"
Sep 17 21:28:44.573179 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.573159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:44.577991 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.577967    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-b25p2\""
Sep 17 21:28:44.578888 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.578858    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-slzmb\""
Sep 17 21:28:44.579027 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.578895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 21:28:44.579217 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.579199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 21:28:44.579484 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.579468    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 21:28:44.586029 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.586004    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"]
Sep 17 21:28:44.683118 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.683084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:44.683118 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.683117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fwlz\" (UniqueName: \"kubernetes.io/projected/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-kube-api-access-5fwlz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:44.784331 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.784293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:44.784331 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.784329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fwlz\" (UniqueName: \"kubernetes.io/projected/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-kube-api-access-5fwlz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:44.786935 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.786904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:44.793266 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.793225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fwlz\" (UniqueName: \"kubernetes.io/projected/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-kube-api-access-5fwlz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:44.883300 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:44.883195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:45.166911 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:45.166843    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"]
Sep 17 21:28:45.169367 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:28:45.169337    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e1e36f_f78f_4a37_b0e2_ce8f8c9d1e27.slice/crio-1d3fbad83d61a864baf9863a5abd3d00d55b2d77426a9e3446752c877ea1b6eb WatchSource:0}: Error finding container 1d3fbad83d61a864baf9863a5abd3d00d55b2d77426a9e3446752c877ea1b6eb: Status 404 returned error can't find the container with id 1d3fbad83d61a864baf9863a5abd3d00d55b2d77426a9e3446752c877ea1b6eb
Sep 17 21:28:45.644694 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:45.644655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" event={"ID":"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27","Type":"ContainerStarted","Data":"7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c"}
Sep 17 21:28:45.645145 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:45.644710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" event={"ID":"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27","Type":"ContainerStarted","Data":"1d3fbad83d61a864baf9863a5abd3d00d55b2d77426a9e3446752c877ea1b6eb"}
Sep 17 21:28:45.645145 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:45.644731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:28:45.662582 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:28:45.662533    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" podStartSLOduration=1.662519482 podStartE2EDuration="1.662519482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:28:45.660469676 +0000 UTC m=+2364.479761639" watchObservedRunningTime="2026-09-17 21:28:45.662519482 +0000 UTC m=+2364.481811468"
Sep 17 21:29:16.650868 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:29:16.650836    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:29:56.980383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:29:56.980349    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"]
Sep 17 21:30:04.396841 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:04.396802    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:30:12.000928 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:12.000873    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main" containerID="cri-o://da40fc9ad9fe834a43f4fde84c529b2fc5ecdf1f0dbd19b0c61584ffbf26f3c1" gracePeriod=45
Sep 17 21:30:14.396524 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:14.396483    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:30:24.397391 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:24.397350    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:30:24.397772 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:24.397452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:30:31.809646 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:31.809612    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"]
Sep 17 21:30:34.396875 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:34.396830    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:30:36.651100 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:36.651060    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:30:42.202608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.202574    2577 generic.go:356] "Generic (PLEG): container finished" podID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerID="da40fc9ad9fe834a43f4fde84c529b2fc5ecdf1f0dbd19b0c61584ffbf26f3c1" exitCode=0
Sep 17 21:30:42.203007 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.202652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" event={"ID":"15fca102-3e69-43ab-a17a-8d4cfd22e90a","Type":"ContainerDied","Data":"da40fc9ad9fe834a43f4fde84c529b2fc5ecdf1f0dbd19b0c61584ffbf26f3c1"}
Sep 17 21:30:42.242980 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.242953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:30:42.332376 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.332282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6kbf\" (UniqueName: \"kubernetes.io/projected/15fca102-3e69-43ab-a17a-8d4cfd22e90a-kube-api-access-n6kbf\") pod \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") "
Sep 17 21:30:42.332376 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.332323    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fca102-3e69-43ab-a17a-8d4cfd22e90a-tls-certs\") pod \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\" (UID: \"15fca102-3e69-43ab-a17a-8d4cfd22e90a\") "
Sep 17 21:30:42.334984 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.334950    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15fca102-3e69-43ab-a17a-8d4cfd22e90a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15fca102-3e69-43ab-a17a-8d4cfd22e90a" (UID: "15fca102-3e69-43ab-a17a-8d4cfd22e90a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:30:42.335120 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.334994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fca102-3e69-43ab-a17a-8d4cfd22e90a-kube-api-access-n6kbf" (OuterVolumeSpecName: "kube-api-access-n6kbf") pod "15fca102-3e69-43ab-a17a-8d4cfd22e90a" (UID: "15fca102-3e69-43ab-a17a-8d4cfd22e90a"). InnerVolumeSpecName "kube-api-access-n6kbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:30:42.433111 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.433075    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6kbf\" (UniqueName: \"kubernetes.io/projected/15fca102-3e69-43ab-a17a-8d4cfd22e90a-kube-api-access-n6kbf\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:30:42.433111 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:42.433107    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fca102-3e69-43ab-a17a-8d4cfd22e90a-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:30:43.208366 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:43.208331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx" event={"ID":"15fca102-3e69-43ab-a17a-8d4cfd22e90a","Type":"ContainerDied","Data":"7df603200cd7b75e8bbcef078a3569fccb2c22306f83871c241561bd4f0e426e"}
Sep 17 21:30:43.208366 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:43.208370    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"
Sep 17 21:30:43.208887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:43.208393    2577 scope.go:117] "RemoveContainer" containerID="da40fc9ad9fe834a43f4fde84c529b2fc5ecdf1f0dbd19b0c61584ffbf26f3c1"
Sep 17 21:30:43.229939 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:43.229916    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"]
Sep 17 21:30:43.235968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:43.235945    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58fvxnx"]
Sep 17 21:30:43.717573 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:43.717540    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" path="/var/lib/kubelet/pods/15fca102-3e69-43ab-a17a-8d4cfd22e90a/volumes"
Sep 17 21:30:46.650951 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:46.650916    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:30:46.829233 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:46.829188    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main" containerID="cri-o://7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c" gracePeriod=45
Sep 17 21:30:47.684781 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.684723    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"]
Sep 17 21:30:47.685250 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.685056    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main"
Sep 17 21:30:47.685250 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.685068    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main"
Sep 17 21:30:47.685250 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.685120    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fca102-3e69-43ab-a17a-8d4cfd22e90a" containerName="main"
Sep 17 21:30:47.689537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.689518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.700995 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.700970    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 21:30:47.701113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.701079    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 21:30:47.701113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.701095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 21:30:47.701273 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.701229    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-pm56r\""
Sep 17 21:30:47.718956 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.718930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"]
Sep 17 21:30:47.779341 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-token\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-credential-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-envoy\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-podinfo\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779761 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-data\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779761 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-certs\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779761 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ff42\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-kube-api-access-5ff42\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.779761 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.779651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cac34a6-f617-4996-bbcf-cbac897e0845-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.880858 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.880820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-token\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.880871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-credential-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.880894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-envoy\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-podinfo\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881341 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881341 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-data\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881341 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-certs\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881341 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ff42\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-kube-api-access-5ff42\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881341 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cac34a6-f617-4996-bbcf-cbac897e0845-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881612 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-credential-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881695 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881749 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-certs\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881819 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-data\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.881993 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.881976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cac34a6-f617-4996-bbcf-cbac897e0845-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.883471 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.883448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-envoy\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.883581 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.883565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-podinfo\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.895432 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.895368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-token\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.898521 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.898503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ff42\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-kube-api-access-5ff42\") pod \"router-gateway-1-openshift-default-544949f8b8-cjzp9\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:47.999610 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:47.999580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:48.283747 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:48.283723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"]
Sep 17 21:30:48.286059 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:30:48.286030    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cac34a6_f617_4996_bbcf_cbac897e0845.slice/crio-8d7ca137419b73d255c0deef815764e9d37fea8df5374e74ecc203a31815433c WatchSource:0}: Error finding container 8d7ca137419b73d255c0deef815764e9d37fea8df5374e74ecc203a31815433c: Status 404 returned error can't find the container with id 8d7ca137419b73d255c0deef815764e9d37fea8df5374e74ecc203a31815433c
Sep 17 21:30:48.288156 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:48.288115    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:30:48.288266 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:48.288190    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:30:48.288266 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:48.288222    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:30:49.240090 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:49.240042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" event={"ID":"3cac34a6-f617-4996-bbcf-cbac897e0845","Type":"ContainerStarted","Data":"657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635"}
Sep 17 21:30:49.240617 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:49.240122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" event={"ID":"3cac34a6-f617-4996-bbcf-cbac897e0845","Type":"ContainerStarted","Data":"8d7ca137419b73d255c0deef815764e9d37fea8df5374e74ecc203a31815433c"}
Sep 17 21:30:49.278252 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:49.278176    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" podStartSLOduration=2.278158838 podStartE2EDuration="2.278158838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:30:49.277058438 +0000 UTC m=+2488.096350401" watchObservedRunningTime="2026-09-17 21:30:49.278158838 +0000 UTC m=+2488.097450800"
Sep 17 21:30:50.000198 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:50.000155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:50.001643 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:50.001611    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 17 21:30:51.000294 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:51.000256    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 17 21:30:52.000485 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:52.000437    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 17 21:30:53.000332 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:53.000287    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 17 21:30:54.000411 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:54.000371    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 17 21:30:55.004696 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:55.004662    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:55.005193 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:55.005032    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:55.005577 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:55.005558    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:30:56.651976 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:56.651931    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:30:56.652503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:30:56.652040    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:31:06.651258 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:06.651196    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:31:16.650930 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:16.650828    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:31:17.085140 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.085112    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:31:17.238643 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.238548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fwlz\" (UniqueName: \"kubernetes.io/projected/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-kube-api-access-5fwlz\") pod \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") "
Sep 17 21:31:17.238643 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.238607    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-tls-certs\") pod \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\" (UID: \"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27\") "
Sep 17 21:31:17.241058 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.241030    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-kube-api-access-5fwlz" (OuterVolumeSpecName: "kube-api-access-5fwlz") pod "e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" (UID: "e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27"). InnerVolumeSpecName "kube-api-access-5fwlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:31:17.241185 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.241092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" (UID: "e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:31:17.339605 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.339564    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:31:17.339605 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.339594    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fwlz\" (UniqueName: \"kubernetes.io/projected/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27-kube-api-access-5fwlz\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:31:17.400250 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.400212    2577 generic.go:356] "Generic (PLEG): container finished" podID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerID="7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c" exitCode=0
Sep 17 21:31:17.400429 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.400278    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"
Sep 17 21:31:17.400429 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.400282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" event={"ID":"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27","Type":"ContainerDied","Data":"7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c"}
Sep 17 21:31:17.400518 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.400441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww" event={"ID":"e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27","Type":"ContainerDied","Data":"1d3fbad83d61a864baf9863a5abd3d00d55b2d77426a9e3446752c877ea1b6eb"}
Sep 17 21:31:17.400518 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.400468    2577 scope.go:117] "RemoveContainer" containerID="7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c"
Sep 17 21:31:17.410015 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.409998    2577 scope.go:117] "RemoveContainer" containerID="7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c"
Sep 17 21:31:17.410322 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:31:17.410296    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c\": container with ID starting with 7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c not found: ID does not exist" containerID="7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c"
Sep 17 21:31:17.410372 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.410332    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c"} err="failed to get container status \"7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c\": rpc error: code = NotFound desc = could not find container \"7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c\": container with ID starting with 7c681005bed9f6c2e9d1337449c9f4bd5e43913c5ae29da5a91ee7cdceeaf72c not found: ID does not exist"
Sep 17 21:31:17.421949 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.421923    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"]
Sep 17 21:31:17.428171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.428150    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenqqww"]
Sep 17 21:31:17.717702 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:17.717670    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" path="/var/lib/kubelet/pods/e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27/volumes"
Sep 17 21:31:51.836853 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.836817    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"]
Sep 17 21:31:51.837468 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.837325    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main"
Sep 17 21:31:51.837468 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.837345    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main"
Sep 17 21:31:51.837468 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.837431    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e1e36f-f78f-4a37-b0e2-ce8f8c9d1e27" containerName="main"
Sep 17 21:31:51.840579 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.840557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.843019 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.842997    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 21:31:51.843143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.843103    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-l79fg\""
Sep 17 21:31:51.843222 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.843161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 21:31:51.843334 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.843292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 21:31:51.843437 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.843420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:31:51.843696 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.843680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 21:31:51.860399 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.860373    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"]
Sep 17 21:31:51.898333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.898493 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.898493 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadfa822-8057-46fc-b0dc-35e7e0743183-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.898493 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eadfa822-8057-46fc-b0dc-35e7e0743183-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.898493 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.898670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.898670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.898670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.898549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnztx\" (UniqueName: \"kubernetes.io/projected/eadfa822-8057-46fc-b0dc-35e7e0743183-kube-api-access-wnztx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999662 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999662 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnztx\" (UniqueName: \"kubernetes.io/projected/eadfa822-8057-46fc-b0dc-35e7e0743183-kube-api-access-wnztx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999662 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999816 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999816 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadfa822-8057-46fc-b0dc-35e7e0743183-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999816 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eadfa822-8057-46fc-b0dc-35e7e0743183-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999972 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:51.999972 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:51.999947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.000084 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.000046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.000145 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.000088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.000400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.000383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eadfa822-8057-46fc-b0dc-35e7e0743183-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.001929 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.001906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.002324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.002304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadfa822-8057-46fc-b0dc-35e7e0743183-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.009439 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.009417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnztx\" (UniqueName: \"kubernetes.io/projected/eadfa822-8057-46fc-b0dc-35e7e0743183-kube-api-access-wnztx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.104835 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.104743    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"]
Sep 17 21:31:52.109148 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.109125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.111786 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.111765    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-9f9qr\""
Sep 17 21:31:52.125538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.125515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"]
Sep 17 21:31:52.150474 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.150446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:31:52.201512 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.201467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpw2\" (UniqueName: \"kubernetes.io/projected/7a052b84-a1fb-47b3-adbc-4a0932624aed-kube-api-access-dvpw2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.202257 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.202184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a052b84-a1fb-47b3-adbc-4a0932624aed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.302939 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.302906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpw2\" (UniqueName: \"kubernetes.io/projected/7a052b84-a1fb-47b3-adbc-4a0932624aed-kube-api-access-dvpw2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.303110 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.302970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a052b84-a1fb-47b3-adbc-4a0932624aed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.305578 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.305550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a052b84-a1fb-47b3-adbc-4a0932624aed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.313400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.313376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpw2\" (UniqueName: \"kubernetes.io/projected/7a052b84-a1fb-47b3-adbc-4a0932624aed-kube-api-access-dvpw2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.419424 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.419351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:52.438154 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.438123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"]
Sep 17 21:31:52.441779 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:31:52.441736    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadfa822_8057_46fc_b0dc_35e7e0743183.slice/crio-16a82318b2d5442dd970b403fe1af6f4fafdd0205f0a2ab5edd9aa1cf6434fb8 WatchSource:0}: Error finding container 16a82318b2d5442dd970b403fe1af6f4fafdd0205f0a2ab5edd9aa1cf6434fb8: Status 404 returned error can't find the container with id 16a82318b2d5442dd970b403fe1af6f4fafdd0205f0a2ab5edd9aa1cf6434fb8
Sep 17 21:31:52.564365 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.564331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" event={"ID":"eadfa822-8057-46fc-b0dc-35e7e0743183","Type":"ContainerStarted","Data":"4e94b734388b0999b66395c9c875dd503da9f29430fd54b19de5f56287417eb0"}
Sep 17 21:31:52.564525 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.564385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" event={"ID":"eadfa822-8057-46fc-b0dc-35e7e0743183","Type":"ContainerStarted","Data":"16a82318b2d5442dd970b403fe1af6f4fafdd0205f0a2ab5edd9aa1cf6434fb8"}
Sep 17 21:31:52.701464 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:52.701386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"]
Sep 17 21:31:52.704779 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:31:52.704744    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a052b84_a1fb_47b3_adbc_4a0932624aed.slice/crio-12742a08ef1fd9726034df9fc4fc0e4775db5cdb4e57fe6b8d0b4cc6b2264248 WatchSource:0}: Error finding container 12742a08ef1fd9726034df9fc4fc0e4775db5cdb4e57fe6b8d0b4cc6b2264248: Status 404 returned error can't find the container with id 12742a08ef1fd9726034df9fc4fc0e4775db5cdb4e57fe6b8d0b4cc6b2264248
Sep 17 21:31:53.572160 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:53.572121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" event={"ID":"7a052b84-a1fb-47b3-adbc-4a0932624aed","Type":"ContainerStarted","Data":"0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14"}
Sep 17 21:31:53.572705 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:53.572179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" event={"ID":"7a052b84-a1fb-47b3-adbc-4a0932624aed","Type":"ContainerStarted","Data":"12742a08ef1fd9726034df9fc4fc0e4775db5cdb4e57fe6b8d0b4cc6b2264248"}
Sep 17 21:31:53.602724 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:53.602645    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" podStartSLOduration=1.602624174 podStartE2EDuration="1.602624174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:31:53.601174914 +0000 UTC m=+2552.420466877" watchObservedRunningTime="2026-09-17 21:31:53.602624174 +0000 UTC m=+2552.421916136"
Sep 17 21:31:54.577482 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:54.577450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:31:55.583976 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:55.583940    2577 generic.go:356] "Generic (PLEG): container finished" podID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerID="4e94b734388b0999b66395c9c875dd503da9f29430fd54b19de5f56287417eb0" exitCode=0
Sep 17 21:31:55.584397 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:55.584012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" event={"ID":"eadfa822-8057-46fc-b0dc-35e7e0743183","Type":"ContainerDied","Data":"4e94b734388b0999b66395c9c875dd503da9f29430fd54b19de5f56287417eb0"}
Sep 17 21:31:55.585438 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:31:55.585423    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:32:24.767441 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:24.767402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" event={"ID":"eadfa822-8057-46fc-b0dc-35e7e0743183","Type":"ContainerStarted","Data":"90521756db3eca2f89dc7d4417b126c85498129ac08b21440dd02023ac5ac2fb"}
Sep 17 21:32:24.797664 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:24.797601    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podStartSLOduration=2.828469734 podStartE2EDuration="33.797582188s" totalImagesPullingTime="28.312079879s" totalInitContainerRuntime="2.657032575s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:32:24.795429541 +0000 UTC m=+2583.614721504" watchObservedRunningTime="2026-09-17 21:32:24.797582188 +0000 UTC m=+2583.616874151"
Sep 17 21:32:25.151547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:25.151459    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:32:25.586126 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:25.586098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:32:32.151680 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:32.151637    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:32:32.153138 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:32.153112    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:32:42.151105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:42.151063    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:32:52.151035 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:32:52.150991    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:33:02.150882 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:02.150842    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:33:12.150877 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:12.150831    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:33:22.151127 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:22.151071    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:33:32.151966 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:32.151921    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:33:42.161605 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:42.161571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:33:42.169176 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:42.169152    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:33:43.768365 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:43.768326    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"]
Sep 17 21:33:43.795653 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:43.795624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"]
Sep 17 21:33:45.586433 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:45.586389    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:33:55.585788 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:55.585749    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:33:58.787495 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:58.787433    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main" containerID="cri-o://0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14" gracePeriod=45
Sep 17 21:33:58.816983 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:58.816946    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" containerID="cri-o://90521756db3eca2f89dc7d4417b126c85498129ac08b21440dd02023ac5ac2fb" gracePeriod=45
Sep 17 21:33:59.163109 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.163008    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 21:33:59.278801 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.278770    2577 generic.go:356] "Generic (PLEG): container finished" podID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerID="90521756db3eca2f89dc7d4417b126c85498129ac08b21440dd02023ac5ac2fb" exitCode=0
Sep 17 21:33:59.278969 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.278827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" event={"ID":"eadfa822-8057-46fc-b0dc-35e7e0743183","Type":"ContainerDied","Data":"90521756db3eca2f89dc7d4417b126c85498129ac08b21440dd02023ac5ac2fb"}
Sep 17 21:33:59.382162 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.382137    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:33:59.518920 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.518878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-tmp-dir\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.518920 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.518919    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-dshm\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.519184 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.518941    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnztx\" (UniqueName: \"kubernetes.io/projected/eadfa822-8057-46fc-b0dc-35e7e0743183-kube-api-access-wnztx\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.519184 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.518978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-kserve-provision-location\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.519184 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519004    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-model-cache\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.519184 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-home\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.519184 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eadfa822-8057-46fc-b0dc-35e7e0743183-cabundle-cert\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.519477 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadfa822-8057-46fc-b0dc-35e7e0743183-tls-certs\") pod \"eadfa822-8057-46fc-b0dc-35e7e0743183\" (UID: \"eadfa822-8057-46fc-b0dc-35e7e0743183\") "
Sep 17 21:33:59.519477 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-model-cache" (OuterVolumeSpecName: "model-cache") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:59.519477 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:59.519588 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-home" (OuterVolumeSpecName: "home") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:59.519872 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519824    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:33:59.519872 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519852    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:33:59.519872 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519865    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:33:59.520115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.519889    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eadfa822-8057-46fc-b0dc-35e7e0743183-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:33:59.521526 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.521495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-dshm" (OuterVolumeSpecName: "dshm") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:59.521637 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.521549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadfa822-8057-46fc-b0dc-35e7e0743183-kube-api-access-wnztx" (OuterVolumeSpecName: "kube-api-access-wnztx") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "kube-api-access-wnztx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:33:59.521637 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.521605    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eadfa822-8057-46fc-b0dc-35e7e0743183-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:33:59.538618 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.538595    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eadfa822-8057-46fc-b0dc-35e7e0743183" (UID: "eadfa822-8057-46fc-b0dc-35e7e0743183"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:59.621057 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.621022    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-kserve-provision-location\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:33:59.621057 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.621052    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eadfa822-8057-46fc-b0dc-35e7e0743183-cabundle-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:33:59.621057 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.621062    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadfa822-8057-46fc-b0dc-35e7e0743183-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:33:59.621295 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.621071    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eadfa822-8057-46fc-b0dc-35e7e0743183-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:33:59.621295 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:33:59.621079    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnztx\" (UniqueName: \"kubernetes.io/projected/eadfa822-8057-46fc-b0dc-35e7e0743183-kube-api-access-wnztx\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:00.285728 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:00.285687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z" event={"ID":"eadfa822-8057-46fc-b0dc-35e7e0743183","Type":"ContainerDied","Data":"16a82318b2d5442dd970b403fe1af6f4fafdd0205f0a2ab5edd9aa1cf6434fb8"}
Sep 17 21:34:00.286179 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:00.285760    2577 scope.go:117] "RemoveContainer" containerID="90521756db3eca2f89dc7d4417b126c85498129ac08b21440dd02023ac5ac2fb"
Sep 17 21:34:00.286179 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:00.285705    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"
Sep 17 21:34:00.294312 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:00.294173    2577 scope.go:117] "RemoveContainer" containerID="4e94b734388b0999b66395c9c875dd503da9f29430fd54b19de5f56287417eb0"
Sep 17 21:34:00.306369 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:00.306348    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"]
Sep 17 21:34:00.312587 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:00.312568    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-74887b5bfd-shg4z"]
Sep 17 21:34:01.717433 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:01.717392    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" path="/var/lib/kubelet/pods/eadfa822-8057-46fc-b0dc-35e7e0743183/volumes"
Sep 17 21:34:05.585839 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:05.585799    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:34:05.586355 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:05.585914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:34:10.068793 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:10.068758    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"]
Sep 17 21:34:10.069268 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:10.068943    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy" containerID="cri-o://657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635" gracePeriod=30
Sep 17 21:34:14.136694 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.136618    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"]
Sep 17 21:34:14.137048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.136939    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main"
Sep 17 21:34:14.137048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.136951    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main"
Sep 17 21:34:14.137048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.136962    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="storage-initializer"
Sep 17 21:34:14.137048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.136968    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="storage-initializer"
Sep 17 21:34:14.137048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.137029    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eadfa822-8057-46fc-b0dc-35e7e0743183" containerName="main"
Sep 17 21:34:14.141660 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.141638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.146406 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.146385    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-h4ldl\""
Sep 17 21:34:14.146762 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.146740    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 21:34:14.146841 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.146821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 21:34:14.146979 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.146950    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 21:34:14.147042 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.146982    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-wmnwr\""
Sep 17 21:34:14.161033 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.161009    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"]
Sep 17 21:34:14.243073 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.243025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxlj\" (UniqueName: \"kubernetes.io/projected/870e203e-838b-4464-98e2-fcd6dd852ac9-kube-api-access-llxlj\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.243262 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.243164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e203e-838b-4464-98e2-fcd6dd852ac9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.344020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.343991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e203e-838b-4464-98e2-fcd6dd852ac9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.344196 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.344039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxlj\" (UniqueName: \"kubernetes.io/projected/870e203e-838b-4464-98e2-fcd6dd852ac9-kube-api-access-llxlj\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.346970 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.346942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e203e-838b-4464-98e2-fcd6dd852ac9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.356743 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.356709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxlj\" (UniqueName: \"kubernetes.io/projected/870e203e-838b-4464-98e2-fcd6dd852ac9-kube-api-access-llxlj\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.452538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.452448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:14.762691 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:34:14.762662    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870e203e_838b_4464_98e2_fcd6dd852ac9.slice/crio-af73ac6add610de32c2cb488419c32368be5a57565f9e6719dd29f9489aa2b62 WatchSource:0}: Error finding container af73ac6add610de32c2cb488419c32368be5a57565f9e6719dd29f9489aa2b62: Status 404 returned error can't find the container with id af73ac6add610de32c2cb488419c32368be5a57565f9e6719dd29f9489aa2b62
Sep 17 21:34:14.763088 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:14.763063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"]
Sep 17 21:34:15.312809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.312781    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:34:15.359973 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.359925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" event={"ID":"870e203e-838b-4464-98e2-fcd6dd852ac9","Type":"ContainerStarted","Data":"3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f"}
Sep 17 21:34:15.360143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.360001    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:15.360143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.360026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" event={"ID":"870e203e-838b-4464-98e2-fcd6dd852ac9","Type":"ContainerStarted","Data":"af73ac6add610de32c2cb488419c32368be5a57565f9e6719dd29f9489aa2b62"}
Sep 17 21:34:15.361559 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.361533    2577 generic.go:356] "Generic (PLEG): container finished" podID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerID="657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635" exitCode=0
Sep 17 21:34:15.361705 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.361573    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"
Sep 17 21:34:15.361705 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.361606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" event={"ID":"3cac34a6-f617-4996-bbcf-cbac897e0845","Type":"ContainerDied","Data":"657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635"}
Sep 17 21:34:15.361705 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.361652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9" event={"ID":"3cac34a6-f617-4996-bbcf-cbac897e0845","Type":"ContainerDied","Data":"8d7ca137419b73d255c0deef815764e9d37fea8df5374e74ecc203a31815433c"}
Sep 17 21:34:15.361705 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.361682    2577 scope.go:117] "RemoveContainer" containerID="657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635"
Sep 17 21:34:15.370779 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.370758    2577 scope.go:117] "RemoveContainer" containerID="657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635"
Sep 17 21:34:15.371086 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:34:15.371069    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635\": container with ID starting with 657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635 not found: ID does not exist" containerID="657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635"
Sep 17 21:34:15.371143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.371095    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635"} err="failed to get container status \"657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635\": rpc error: code = NotFound desc = could not find container \"657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635\": container with ID starting with 657136e5357b902c96029ebbb7a8ef26cec6b694e83a955090ee7bf3ce1d9635 not found: ID does not exist"
Sep 17 21:34:15.394191 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.394143    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" podStartSLOduration=1.394127466 podStartE2EDuration="1.394127466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:34:15.392991612 +0000 UTC m=+2694.212283576" watchObservedRunningTime="2026-09-17 21:34:15.394127466 +0000 UTC m=+2694.213419464"
Sep 17 21:34:15.451832 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.451800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-data\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.451832 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.451835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-credential-socket\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452030 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.451861    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cac34a6-f617-4996-bbcf-cbac897e0845-istiod-ca-cert\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452030 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.451888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ff42\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-kube-api-access-5ff42\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452030 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-socket\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452186 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452032    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-token\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452186 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-envoy\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452186 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-data" (OuterVolumeSpecName: "istio-data") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:34:15.452186 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452124    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:34:15.452186 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-podinfo\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452209    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-certs\") pod \"3cac34a6-f617-4996-bbcf-cbac897e0845\" (UID: \"3cac34a6-f617-4996-bbcf-cbac897e0845\") "
Sep 17 21:34:15.452480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cac34a6-f617-4996-bbcf-cbac897e0845-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:34:15.452480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452375    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:34:15.452698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452563    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.452698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452584    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-data\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.452698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452595    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-credential-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.452698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452603    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cac34a6-f617-4996-bbcf-cbac897e0845-istiod-ca-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.452698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.452562    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:34:15.454632 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.454607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-token" (OuterVolumeSpecName: "istio-token") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:34:15.454632 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.454615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:34:15.454792 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.454727    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-kube-api-access-5ff42" (OuterVolumeSpecName: "kube-api-access-5ff42") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "kube-api-access-5ff42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:34:15.454792 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.454731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3cac34a6-f617-4996-bbcf-cbac897e0845" (UID: "3cac34a6-f617-4996-bbcf-cbac897e0845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 21:34:15.553161 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.553123    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-envoy\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.553161 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.553151    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-podinfo\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.553161 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.553159    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cac34a6-f617-4996-bbcf-cbac897e0845-workload-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.553161 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.553168    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ff42\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-kube-api-access-5ff42\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.553529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.553178    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cac34a6-f617-4996-bbcf-cbac897e0845-istio-token\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.585503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.585468    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:34:15.688836 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.688801    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"]
Sep 17 21:34:15.694417 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.694393    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-cjzp9"]
Sep 17 21:34:15.722523 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:15.722496    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" path="/var/lib/kubelet/pods/3cac34a6-f617-4996-bbcf-cbac897e0845/volumes"
Sep 17 21:34:25.586420 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:25.586377    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:34:29.038090 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.038064    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:34:29.055064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.054995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvpw2\" (UniqueName: \"kubernetes.io/projected/7a052b84-a1fb-47b3-adbc-4a0932624aed-kube-api-access-dvpw2\") pod \"7a052b84-a1fb-47b3-adbc-4a0932624aed\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") "
Sep 17 21:34:29.055177 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.055086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a052b84-a1fb-47b3-adbc-4a0932624aed-tls-certs\") pod \"7a052b84-a1fb-47b3-adbc-4a0932624aed\" (UID: \"7a052b84-a1fb-47b3-adbc-4a0932624aed\") "
Sep 17 21:34:29.057750 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.057576    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a052b84-a1fb-47b3-adbc-4a0932624aed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a052b84-a1fb-47b3-adbc-4a0932624aed" (UID: "7a052b84-a1fb-47b3-adbc-4a0932624aed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:34:29.058356 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.058324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a052b84-a1fb-47b3-adbc-4a0932624aed-kube-api-access-dvpw2" (OuterVolumeSpecName: "kube-api-access-dvpw2") pod "7a052b84-a1fb-47b3-adbc-4a0932624aed" (UID: "7a052b84-a1fb-47b3-adbc-4a0932624aed"). InnerVolumeSpecName "kube-api-access-dvpw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:34:29.155831 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.155797    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a052b84-a1fb-47b3-adbc-4a0932624aed-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:29.155831 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.155824    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvpw2\" (UniqueName: \"kubernetes.io/projected/7a052b84-a1fb-47b3-adbc-4a0932624aed-kube-api-access-dvpw2\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:34:29.431423 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.431335    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerID="0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14" exitCode=0
Sep 17 21:34:29.431423 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.431399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"
Sep 17 21:34:29.431608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.431423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" event={"ID":"7a052b84-a1fb-47b3-adbc-4a0932624aed","Type":"ContainerDied","Data":"0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14"}
Sep 17 21:34:29.431608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.431484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch" event={"ID":"7a052b84-a1fb-47b3-adbc-4a0932624aed","Type":"ContainerDied","Data":"12742a08ef1fd9726034df9fc4fc0e4775db5cdb4e57fe6b8d0b4cc6b2264248"}
Sep 17 21:34:29.431608 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.431513    2577 scope.go:117] "RemoveContainer" containerID="0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14"
Sep 17 21:34:29.440664 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.440637    2577 scope.go:117] "RemoveContainer" containerID="0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14"
Sep 17 21:34:29.440979 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:34:29.440959    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14\": container with ID starting with 0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14 not found: ID does not exist" containerID="0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14"
Sep 17 21:34:29.441056 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.440985    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14"} err="failed to get container status \"0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14\": rpc error: code = NotFound desc = could not find container \"0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14\": container with ID starting with 0ba8d21be8ec20a5812e054bc829b38c2a6b1f9d7f6ba5a9982ae765e3deeb14 not found: ID does not exist"
Sep 17 21:34:29.462038 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.462009    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"]
Sep 17 21:34:29.463718 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.463699    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schej2pch"]
Sep 17 21:34:29.717418 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:29.717333    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" path="/var/lib/kubelet/pods/7a052b84-a1fb-47b3-adbc-4a0932624aed/volumes"
Sep 17 21:34:41.509647 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.509365    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"]
Sep 17 21:34:41.510113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.509825    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main"
Sep 17 21:34:41.510113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.509840    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main"
Sep 17 21:34:41.510113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.509866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy"
Sep 17 21:34:41.510113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.509874    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy"
Sep 17 21:34:41.510113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.509932    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cac34a6-f617-4996-bbcf-cbac897e0845" containerName="istio-proxy"
Sep 17 21:34:41.510113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.509944    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a052b84-a1fb-47b3-adbc-4a0932624aed" containerName="main"
Sep 17 21:34:41.513090 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.513074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.515595 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.515566    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:34:41.515722 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.515669    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-q56p2\""
Sep 17 21:34:41.515924 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.515910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 21:34:41.516175 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.516159    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 21:34:41.516359 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.516343    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 21:34:41.516722 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.516708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 21:34:41.533645 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.533614    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"]
Sep 17 21:34:41.555333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a575e1-ed49-4252-91b0-c9570e59b477-tls-certs\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.555509 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-tmp-dir\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.555509 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-kserve-provision-location\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.555677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-dshm\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.555677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-home\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.555677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-model-cache\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.555677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43a575e1-ed49-4252-91b0-c9570e59b477-cabundle-cert\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.555677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.555676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lzm\" (UniqueName: \"kubernetes.io/projected/43a575e1-ed49-4252-91b0-c9570e59b477-kube-api-access-b9lzm\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656027 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.655992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-tmp-dir\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656205 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-kserve-provision-location\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656205 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-dshm\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656205 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-home\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656205 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-model-cache\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656466 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43a575e1-ed49-4252-91b0-c9570e59b477-cabundle-cert\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656466 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lzm\" (UniqueName: \"kubernetes.io/projected/43a575e1-ed49-4252-91b0-c9570e59b477-kube-api-access-b9lzm\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656466 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a575e1-ed49-4252-91b0-c9570e59b477-tls-certs\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656466 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-home\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656676 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-kserve-provision-location\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656676 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-tmp-dir\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656676 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-model-cache\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.656912 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.656891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43a575e1-ed49-4252-91b0-c9570e59b477-cabundle-cert\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.658438 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.658422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-dshm\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.659084 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.659060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a575e1-ed49-4252-91b0-c9570e59b477-tls-certs\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.664622 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.664603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lzm\" (UniqueName: \"kubernetes.io/projected/43a575e1-ed49-4252-91b0-c9570e59b477-kube-api-access-b9lzm\") pod \"stop-feature-test-kserve-555fc44b76-j7mlm\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:41.822898 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:41.822810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:42.111275 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:42.111230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"]
Sep 17 21:34:42.112786 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:34:42.112764    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a575e1_ed49_4252_91b0_c9570e59b477.slice/crio-df04403de21c1bb86f5fc9251e485cd43d1299d5497efe0345797aeedcd0c08e WatchSource:0}: Error finding container df04403de21c1bb86f5fc9251e485cd43d1299d5497efe0345797aeedcd0c08e: Status 404 returned error can't find the container with id df04403de21c1bb86f5fc9251e485cd43d1299d5497efe0345797aeedcd0c08e
Sep 17 21:34:42.494677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:42.494581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" event={"ID":"43a575e1-ed49-4252-91b0-c9570e59b477","Type":"ContainerStarted","Data":"d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591"}
Sep 17 21:34:42.494677 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:42.494675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" event={"ID":"43a575e1-ed49-4252-91b0-c9570e59b477","Type":"ContainerStarted","Data":"df04403de21c1bb86f5fc9251e485cd43d1299d5497efe0345797aeedcd0c08e"}
Sep 17 21:34:44.504928 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:44.504890    2577 generic.go:356] "Generic (PLEG): container finished" podID="43a575e1-ed49-4252-91b0-c9570e59b477" containerID="d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591" exitCode=0
Sep 17 21:34:44.505433 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:44.504965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" event={"ID":"43a575e1-ed49-4252-91b0-c9570e59b477","Type":"ContainerDied","Data":"d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591"}
Sep 17 21:34:45.512538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:45.512496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" event={"ID":"43a575e1-ed49-4252-91b0-c9570e59b477","Type":"ContainerStarted","Data":"f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252"}
Sep 17 21:34:45.539628 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:45.539561    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podStartSLOduration=2.411583598 podStartE2EDuration="4.539541646s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.127958048s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:34:45.537390017 +0000 UTC m=+2724.356681979" watchObservedRunningTime="2026-09-17 21:34:45.539541646 +0000 UTC m=+2724.358833609"
Sep 17 21:34:45.823468 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:45.823383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:46.371389 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:46.371356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:34:51.823373 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:51.823336    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:34:51.824912 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:34:51.824885    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:35:01.823778 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:35:01.823731    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:35:11.824021 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:35:11.823973    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:35:21.823833 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:35:21.823788    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:35:31.824064 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:35:31.824017    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:35:41.823525 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:35:41.823477    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:35:51.823596 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:35:51.823549    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:36:01.824088 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:01.824039    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:36:11.823806 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:11.823761    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:36:21.824303 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:21.824226    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:36:23.148156 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:23.148076    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"]
Sep 17 21:36:26.371641 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:26.371599    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:36:31.823587 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:31.823541    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:36:36.371562 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:36.371522    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:36:38.168523 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:38.168483    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main" containerID="cri-o://3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f" gracePeriod=45
Sep 17 21:36:41.833227 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:41.833192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:36:41.841063 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:41.841035    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:36:43.695209 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:43.695171    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"]
Sep 17 21:36:46.371162 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:46.371122    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:36:46.371615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:46.371277    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:36:56.371646 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:56.371601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:36:58.718005 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:58.717936    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" containerID="cri-o://f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252" gracePeriod=45
Sep 17 21:36:58.833896 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:58.833848    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 21:36:59.464517 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.464494    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:36:59.595179 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-tmp-dir\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595179 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-kserve-provision-location\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595469 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a575e1-ed49-4252-91b0-c9570e59b477-tls-certs\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595469 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9lzm\" (UniqueName: \"kubernetes.io/projected/43a575e1-ed49-4252-91b0-c9570e59b477-kube-api-access-b9lzm\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595469 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43a575e1-ed49-4252-91b0-c9570e59b477-cabundle-cert\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595469 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595317    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-model-cache\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595469 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-home\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595469 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-dshm\") pod \"43a575e1-ed49-4252-91b0-c9570e59b477\" (UID: \"43a575e1-ed49-4252-91b0-c9570e59b477\") "
Sep 17 21:36:59.595781 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-model-cache" (OuterVolumeSpecName: "model-cache") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:59.595837 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.595712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a575e1-ed49-4252-91b0-c9570e59b477-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:36:59.596361 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.596333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-home" (OuterVolumeSpecName: "home") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:59.597793 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.597764    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a575e1-ed49-4252-91b0-c9570e59b477-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:36:59.597903 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.597812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-dshm" (OuterVolumeSpecName: "dshm") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:59.597963 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.597921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a575e1-ed49-4252-91b0-c9570e59b477-kube-api-access-b9lzm" (OuterVolumeSpecName: "kube-api-access-b9lzm") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "kube-api-access-b9lzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:36:59.608256 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.608211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:59.618511 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.618480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43a575e1-ed49-4252-91b0-c9570e59b477" (UID: "43a575e1-ed49-4252-91b0-c9570e59b477"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:59.696048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696015    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:36:59.696048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696044    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:36:59.696048 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696053    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:36:59.696308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696061    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:36:59.696308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696072    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43a575e1-ed49-4252-91b0-c9570e59b477-kserve-provision-location\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:36:59.696308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696080    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a575e1-ed49-4252-91b0-c9570e59b477-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:36:59.696308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696090    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9lzm\" (UniqueName: \"kubernetes.io/projected/43a575e1-ed49-4252-91b0-c9570e59b477-kube-api-access-b9lzm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:36:59.696308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:36:59.696098    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43a575e1-ed49-4252-91b0-c9570e59b477-cabundle-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:37:00.210069 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.210027    2577 generic.go:356] "Generic (PLEG): container finished" podID="43a575e1-ed49-4252-91b0-c9570e59b477" containerID="f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252" exitCode=0
Sep 17 21:37:00.210537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.210082    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"
Sep 17 21:37:00.210537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.210113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" event={"ID":"43a575e1-ed49-4252-91b0-c9570e59b477","Type":"ContainerDied","Data":"f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252"}
Sep 17 21:37:00.210537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.210189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm" event={"ID":"43a575e1-ed49-4252-91b0-c9570e59b477","Type":"ContainerDied","Data":"df04403de21c1bb86f5fc9251e485cd43d1299d5497efe0345797aeedcd0c08e"}
Sep 17 21:37:00.210537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.210211    2577 scope.go:117] "RemoveContainer" containerID="f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252"
Sep 17 21:37:00.218858 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.218843    2577 scope.go:117] "RemoveContainer" containerID="d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591"
Sep 17 21:37:00.226771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.226750    2577 scope.go:117] "RemoveContainer" containerID="f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252"
Sep 17 21:37:00.227085 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:37:00.227056    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252\": container with ID starting with f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252 not found: ID does not exist" containerID="f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252"
Sep 17 21:37:00.227197 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.227101    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252"} err="failed to get container status \"f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252\": rpc error: code = NotFound desc = could not find container \"f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252\": container with ID starting with f4de4c716894c768fb8e47783f62b38b7f011bd9790fe7dc845341795e664252 not found: ID does not exist"
Sep 17 21:37:00.227197 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.227127    2577 scope.go:117] "RemoveContainer" containerID="d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591"
Sep 17 21:37:00.227458 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:37:00.227438    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591\": container with ID starting with d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591 not found: ID does not exist" containerID="d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591"
Sep 17 21:37:00.227537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.227465    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591"} err="failed to get container status \"d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591\": rpc error: code = NotFound desc = could not find container \"d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591\": container with ID starting with d509b15b76ca515b80694b37ff82a50308b0da9b6065d62ab2b0e0b6175eb591 not found: ID does not exist"
Sep 17 21:37:00.230253 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.230221    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"]
Sep 17 21:37:00.233929 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.233910    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-j7mlm"]
Sep 17 21:37:00.588822 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.588780    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"]
Sep 17 21:37:00.589099 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.589085    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main"
Sep 17 21:37:00.589099 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.589098    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main"
Sep 17 21:37:00.589274 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.589106    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="storage-initializer"
Sep 17 21:37:00.589274 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.589112    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="storage-initializer"
Sep 17 21:37:00.589274 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.589161    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" containerName="main"
Sep 17 21:37:00.592075 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.592052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:00.594711 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.594689    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-q56p2\""
Sep 17 21:37:00.594840 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.594689    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 21:37:00.594840 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.594712    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-4f5ml\""
Sep 17 21:37:00.594979 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.594724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 21:37:00.594979 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.594730    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 21:37:00.603750 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.603725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"]
Sep 17 21:37:00.704805 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.704763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77d1355-634a-4520-84ac-ebdf8d24cbde-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:00.704995 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.704825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj75d\" (UniqueName: \"kubernetes.io/projected/a77d1355-634a-4520-84ac-ebdf8d24cbde-kube-api-access-rj75d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:00.806218 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.806179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj75d\" (UniqueName: \"kubernetes.io/projected/a77d1355-634a-4520-84ac-ebdf8d24cbde-kube-api-access-rj75d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:00.806394 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.806258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77d1355-634a-4520-84ac-ebdf8d24cbde-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:00.808878 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.808855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77d1355-634a-4520-84ac-ebdf8d24cbde-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:00.813840 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.813812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj75d\" (UniqueName: \"kubernetes.io/projected/a77d1355-634a-4520-84ac-ebdf8d24cbde-kube-api-access-rj75d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:00.904522 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:00.904453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:01.177437 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:01.177410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"]
Sep 17 21:37:01.179365 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:37:01.179338    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77d1355_634a_4520_84ac_ebdf8d24cbde.slice/crio-de35e0451612a1ae68543ce934e0348571d24eb65c2601dc17f177f4078097e2 WatchSource:0}: Error finding container de35e0451612a1ae68543ce934e0348571d24eb65c2601dc17f177f4078097e2: Status 404 returned error can't find the container with id de35e0451612a1ae68543ce934e0348571d24eb65c2601dc17f177f4078097e2
Sep 17 21:37:01.216340 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:01.216228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" event={"ID":"a77d1355-634a-4520-84ac-ebdf8d24cbde","Type":"ContainerStarted","Data":"de35e0451612a1ae68543ce934e0348571d24eb65c2601dc17f177f4078097e2"}
Sep 17 21:37:01.717811 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:01.717774    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43a575e1-ed49-4252-91b0-c9570e59b477" path="/var/lib/kubelet/pods/43a575e1-ed49-4252-91b0-c9570e59b477/volumes"
Sep 17 21:37:02.222580 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:02.222542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" event={"ID":"a77d1355-634a-4520-84ac-ebdf8d24cbde","Type":"ContainerStarted","Data":"7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab"}
Sep 17 21:37:02.223085 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:02.222643    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:02.242226 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:02.242174    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" podStartSLOduration=2.242159168 podStartE2EDuration="2.242159168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:37:02.239068433 +0000 UTC m=+2861.058360394" watchObservedRunningTime="2026-09-17 21:37:02.242159168 +0000 UTC m=+2861.061451130"
Sep 17 21:37:06.370922 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:06.370881    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:37:08.413450 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:08.413424    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:37:08.569670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:08.569642    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llxlj\" (UniqueName: \"kubernetes.io/projected/870e203e-838b-4464-98e2-fcd6dd852ac9-kube-api-access-llxlj\") pod \"870e203e-838b-4464-98e2-fcd6dd852ac9\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") "
Sep 17 21:37:08.569670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:08.569678    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e203e-838b-4464-98e2-fcd6dd852ac9-tls-certs\") pod \"870e203e-838b-4464-98e2-fcd6dd852ac9\" (UID: \"870e203e-838b-4464-98e2-fcd6dd852ac9\") "
Sep 17 21:37:08.572068 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:08.572041    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e203e-838b-4464-98e2-fcd6dd852ac9-kube-api-access-llxlj" (OuterVolumeSpecName: "kube-api-access-llxlj") pod "870e203e-838b-4464-98e2-fcd6dd852ac9" (UID: "870e203e-838b-4464-98e2-fcd6dd852ac9"). InnerVolumeSpecName "kube-api-access-llxlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:37:08.572171 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:08.572075    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870e203e-838b-4464-98e2-fcd6dd852ac9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "870e203e-838b-4464-98e2-fcd6dd852ac9" (UID: "870e203e-838b-4464-98e2-fcd6dd852ac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:37:08.670737 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:08.670705    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llxlj\" (UniqueName: \"kubernetes.io/projected/870e203e-838b-4464-98e2-fcd6dd852ac9-kube-api-access-llxlj\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:37:08.670737 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:08.670735    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e203e-838b-4464-98e2-fcd6dd852ac9-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:37:09.256367 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.256334    2577 generic.go:356] "Generic (PLEG): container finished" podID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerID="3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f" exitCode=0
Sep 17 21:37:09.256613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.256385    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"
Sep 17 21:37:09.256613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.256399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" event={"ID":"870e203e-838b-4464-98e2-fcd6dd852ac9","Type":"ContainerDied","Data":"3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f"}
Sep 17 21:37:09.256613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.256438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8" event={"ID":"870e203e-838b-4464-98e2-fcd6dd852ac9","Type":"ContainerDied","Data":"af73ac6add610de32c2cb488419c32368be5a57565f9e6719dd29f9489aa2b62"}
Sep 17 21:37:09.256613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.256459    2577 scope.go:117] "RemoveContainer" containerID="3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f"
Sep 17 21:37:09.265275 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.265233    2577 scope.go:117] "RemoveContainer" containerID="3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f"
Sep 17 21:37:09.265530 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:37:09.265512    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f\": container with ID starting with 3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f not found: ID does not exist" containerID="3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f"
Sep 17 21:37:09.265571 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.265541    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f"} err="failed to get container status \"3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f\": rpc error: code = NotFound desc = could not find container \"3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f\": container with ID starting with 3365fe40ee5163efa70563da9883f29f0aeee74d7efe2984e520ee4776fd826f not found: ID does not exist"
Sep 17 21:37:09.277484 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.277463    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"]
Sep 17 21:37:09.281277 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.281256    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schek9px8"]
Sep 17 21:37:09.717223 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:09.717124    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" path="/var/lib/kubelet/pods/870e203e-838b-4464-98e2-fcd6dd852ac9/volumes"
Sep 17 21:37:23.875586 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.875552    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"]
Sep 17 21:37:23.875989 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.875862    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main"
Sep 17 21:37:23.875989 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.875873    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main"
Sep 17 21:37:23.875989 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.875932    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="870e203e-838b-4464-98e2-fcd6dd852ac9" containerName="main"
Sep 17 21:37:23.880617 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.880600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.886311 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.886292    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:37:23.886577 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.886558    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 21:37:23.886881 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.886852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-4lxl9\""
Sep 17 21:37:23.886881 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.886867    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 21:37:23.887354 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.887016    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-5xggc\""
Sep 17 21:37:23.887354 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.887073    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 21:37:23.887354 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.887128    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 21:37:23.991000 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.990949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.991212 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.991024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-dshm\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.991212 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.991155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-home\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.991212 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.991187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.991422 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.991213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzbf\" (UniqueName: \"kubernetes.io/projected/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kube-api-access-lfzbf\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.991422 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.991267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.991422 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.991299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3c1bed-7286-448d-a3d0-b4e958f4fee1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.991422 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.991339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:23.993396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:23.993364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"]
Sep 17 21:37:24.091997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.091963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-home\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.091997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzbf\" (UniqueName: \"kubernetes.io/projected/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kube-api-access-lfzbf\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3c1bed-7286-448d-a3d0-b4e958f4fee1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-dshm\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-home\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092859 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.092968 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.092947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3c1bed-7286-448d-a3d0-b4e958f4fee1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.095009 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.094989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-dshm\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.095009 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.094998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.105463 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.105428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzbf\" (UniqueName: \"kubernetes.io/projected/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kube-api-access-lfzbf\") pod \"custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.190200 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.190115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:24.252118 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.252020    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"]
Sep 17 21:37:24.257957 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.257574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.261126 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.260937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-hw5mc\""
Sep 17 21:37:24.275957 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.275932    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"]
Sep 17 21:37:24.395069 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.395023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rdq\" (UniqueName: \"kubernetes.io/projected/3336bacd-9167-477d-9e16-cf30be42b4d2-kube-api-access-h8rdq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.395279 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.395099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3336bacd-9167-477d-9e16-cf30be42b4d2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.479388 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.479360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"]
Sep 17 21:37:24.481356 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:37:24.481321    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3c1bed_7286_448d_a3d0_b4e958f4fee1.slice/crio-33b55177fccbfb78b0aa284304e476650a6f24ba9a7df9e190de2d944cdb37d0 WatchSource:0}: Error finding container 33b55177fccbfb78b0aa284304e476650a6f24ba9a7df9e190de2d944cdb37d0: Status 404 returned error can't find the container with id 33b55177fccbfb78b0aa284304e476650a6f24ba9a7df9e190de2d944cdb37d0
Sep 17 21:37:24.483508 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.483487    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:37:24.495586 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.495560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3336bacd-9167-477d-9e16-cf30be42b4d2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.495709 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.495626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rdq\" (UniqueName: \"kubernetes.io/projected/3336bacd-9167-477d-9e16-cf30be42b4d2-kube-api-access-h8rdq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.498160 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.498131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3336bacd-9167-477d-9e16-cf30be42b4d2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.503315 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.503295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rdq\" (UniqueName: \"kubernetes.io/projected/3336bacd-9167-477d-9e16-cf30be42b4d2-kube-api-access-h8rdq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.569483 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.569451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:24.842766 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:24.842737    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"]
Sep 17 21:37:24.844885 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:37:24.844858    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3336bacd_9167_477d_9e16_cf30be42b4d2.slice/crio-e3da5bba8452081f924e796a41a7de7eac513be6a7e97dc849dbcc63d3312e23 WatchSource:0}: Error finding container e3da5bba8452081f924e796a41a7de7eac513be6a7e97dc849dbcc63d3312e23: Status 404 returned error can't find the container with id e3da5bba8452081f924e796a41a7de7eac513be6a7e97dc849dbcc63d3312e23
Sep 17 21:37:25.341425 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:25.341383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" event={"ID":"3336bacd-9167-477d-9e16-cf30be42b4d2","Type":"ContainerStarted","Data":"822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93"}
Sep 17 21:37:25.341919 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:25.341459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" event={"ID":"3336bacd-9167-477d-9e16-cf30be42b4d2","Type":"ContainerStarted","Data":"e3da5bba8452081f924e796a41a7de7eac513be6a7e97dc849dbcc63d3312e23"}
Sep 17 21:37:25.341919 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:25.341497    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:37:25.343039 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:25.343016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerStarted","Data":"33b55177fccbfb78b0aa284304e476650a6f24ba9a7df9e190de2d944cdb37d0"}
Sep 17 21:37:25.363520 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:25.363479    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" podStartSLOduration=1.363465951 podStartE2EDuration="1.363465951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:37:25.361666602 +0000 UTC m=+2884.180958564" watchObservedRunningTime="2026-09-17 21:37:25.363465951 +0000 UTC m=+2884.182757934"
Sep 17 21:37:26.352276 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:26.352204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerStarted","Data":"d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de"}
Sep 17 21:37:26.352920 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:26.352621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:27.360365 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:27.360329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerStarted","Data":"67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a"}
Sep 17 21:37:29.373360 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:29.373319    2577 generic.go:356] "Generic (PLEG): container finished" podID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerID="67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a" exitCode=0
Sep 17 21:37:29.373858 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:29.373371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerDied","Data":"67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a"}
Sep 17 21:37:30.381097 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:30.381061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerStarted","Data":"cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a"}
Sep 17 21:37:30.408101 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:30.408031    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podStartSLOduration=3.644117444 podStartE2EDuration="7.408013976s" totalImagesPullingTime="1.481193991s" totalInitContainerRuntime="2.282702541s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:37:30.406281715 +0000 UTC m=+2889.225573671" watchObservedRunningTime="2026-09-17 21:37:30.408013976 +0000 UTC m=+2889.227305953"
Sep 17 21:37:33.229114 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:33.229084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:37:34.190638 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:34.190598    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:34.190847 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:34.190657    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:34.191990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:34.191956    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:37:44.190845 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:44.190798    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:37:44.203865 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:44.203840    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:37:54.191252 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:54.191190    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:37:56.354383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:37:56.354355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:38:04.191377 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:38:04.191331    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:38:14.190970 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:38:14.190913    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:38:24.190541 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:38:24.190480    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:38:34.191386 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:38:34.191329    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:38:44.190990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:38:44.190930    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:38:54.191188 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:38:54.191140    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:39:02.817367 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:02.817323    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"]
Sep 17 21:39:03.229078 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:03.228991    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:04.190726 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:04.190675    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:39:13.229316 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:13.229276    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:14.190896 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:14.190825    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 21:39:17.838295 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:17.838225    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main" containerID="cri-o://7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab" gracePeriod=45
Sep 17 21:39:23.229636 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:23.229599    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:23.230121 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:23.229718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:39:24.200719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:24.200687    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:39:24.216917 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:24.216891    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:39:33.229290 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:33.229233    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:33.428044 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:33.428009    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"]
Sep 17 21:39:33.432537 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:33.432508    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"]
Sep 17 21:39:36.353906 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:36.353866    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:43.229426 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:43.229382    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:46.354563 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:46.354527    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:48.080206 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.080178    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:39:48.085418 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.085390    2577 generic.go:356] "Generic (PLEG): container finished" podID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerID="7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab" exitCode=0
Sep 17 21:39:48.085557 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.085434    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"
Sep 17 21:39:48.085557 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.085470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" event={"ID":"a77d1355-634a-4520-84ac-ebdf8d24cbde","Type":"ContainerDied","Data":"7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab"}
Sep 17 21:39:48.085557 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.085517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk" event={"ID":"a77d1355-634a-4520-84ac-ebdf8d24cbde","Type":"ContainerDied","Data":"de35e0451612a1ae68543ce934e0348571d24eb65c2601dc17f177f4078097e2"}
Sep 17 21:39:48.085557 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.085539    2577 scope.go:117] "RemoveContainer" containerID="7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab"
Sep 17 21:39:48.093810 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.093793    2577 scope.go:117] "RemoveContainer" containerID="7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab"
Sep 17 21:39:48.094090 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:39:48.094074    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab\": container with ID starting with 7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab not found: ID does not exist" containerID="7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab"
Sep 17 21:39:48.094143 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.094097    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab"} err="failed to get container status \"7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab\": rpc error: code = NotFound desc = could not find container \"7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab\": container with ID starting with 7fc6dbce33acc0000df24a7d21c48a786f5f90ebda6b2efef0dd022c5a5005ab not found: ID does not exist"
Sep 17 21:39:48.095615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.095572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj75d\" (UniqueName: \"kubernetes.io/projected/a77d1355-634a-4520-84ac-ebdf8d24cbde-kube-api-access-rj75d\") pod \"a77d1355-634a-4520-84ac-ebdf8d24cbde\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") "
Sep 17 21:39:48.095684 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.095618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77d1355-634a-4520-84ac-ebdf8d24cbde-tls-certs\") pod \"a77d1355-634a-4520-84ac-ebdf8d24cbde\" (UID: \"a77d1355-634a-4520-84ac-ebdf8d24cbde\") "
Sep 17 21:39:48.097630 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.097609    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77d1355-634a-4520-84ac-ebdf8d24cbde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a77d1355-634a-4520-84ac-ebdf8d24cbde" (UID: "a77d1355-634a-4520-84ac-ebdf8d24cbde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:39:48.097712 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.097698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77d1355-634a-4520-84ac-ebdf8d24cbde-kube-api-access-rj75d" (OuterVolumeSpecName: "kube-api-access-rj75d") pod "a77d1355-634a-4520-84ac-ebdf8d24cbde" (UID: "a77d1355-634a-4520-84ac-ebdf8d24cbde"). InnerVolumeSpecName "kube-api-access-rj75d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:39:48.196449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.196420    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj75d\" (UniqueName: \"kubernetes.io/projected/a77d1355-634a-4520-84ac-ebdf8d24cbde-kube-api-access-rj75d\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:48.196449 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.196444    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77d1355-634a-4520-84ac-ebdf8d24cbde-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:48.409352 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.409320    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"]
Sep 17 21:39:48.413012 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.412989    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-p8kqk"]
Sep 17 21:39:48.452637 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.452582    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main" containerID="cri-o://822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93" gracePeriod=45
Sep 17 21:39:48.455640 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:48.455606    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main" containerID="cri-o://cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a" gracePeriod=45
Sep 17 21:39:49.073448 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.073298    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="llm-d-routing-sidecar" containerID="cri-o://d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de" gracePeriod=45
Sep 17 21:39:49.093372 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.093347    2577 generic.go:356] "Generic (PLEG): container finished" podID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerID="cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a" exitCode=0
Sep 17 21:39:49.093659 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.093398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerDied","Data":"cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a"}
Sep 17 21:39:49.322544 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.322521    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:39:49.406882 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.406849    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3c1bed-7286-448d-a3d0-b4e958f4fee1-cabundle-cert\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407049 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.406912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tmp-dir\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407111 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407090    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-home\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407169 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kserve-provision-location\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407216 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tls-certs\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfzbf\" (UniqueName: \"kubernetes.io/projected/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kube-api-access-lfzbf\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3c1bed-7286-448d-a3d0-b4e958f4fee1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:39:49.407321 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-model-cache\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-dshm\") pod \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\" (UID: \"df3c1bed-7286-448d-a3d0-b4e958f4fee1\") "
Sep 17 21:39:49.407550 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407488    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-model-cache" (OuterVolumeSpecName: "model-cache") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.407768 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407720    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.407768 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.407743    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3c1bed-7286-448d-a3d0-b4e958f4fee1-cabundle-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.408319 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.408292    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-home" (OuterVolumeSpecName: "home") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.409773 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.409749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-dshm" (OuterVolumeSpecName: "dshm") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.409934 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.409774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:39:49.409934 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.409893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kube-api-access-lfzbf" (OuterVolumeSpecName: "kube-api-access-lfzbf") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "kube-api-access-lfzbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:39:49.420271 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.420232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.430346 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.430324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df3c1bed-7286-448d-a3d0-b4e958f4fee1" (UID: "df3c1bed-7286-448d-a3d0-b4e958f4fee1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.509106 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.509078    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.509106 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.509105    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kserve-provision-location\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.509312 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.509116    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.509312 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.509125    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzbf\" (UniqueName: \"kubernetes.io/projected/df3c1bed-7286-448d-a3d0-b4e958f4fee1-kube-api-access-lfzbf\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.509312 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.509133    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.509312 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.509161    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c1bed-7286-448d-a3d0-b4e958f4fee1-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.717254 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:49.717215    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" path="/var/lib/kubelet/pods/a77d1355-634a-4520-84ac-ebdf8d24cbde/volumes"
Sep 17 21:39:50.101063 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.101027    2577 generic.go:356] "Generic (PLEG): container finished" podID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerID="d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de" exitCode=0
Sep 17 21:39:50.101490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.101094    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"
Sep 17 21:39:50.101490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.101099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerDied","Data":"d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de"}
Sep 17 21:39:50.101490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.101157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m" event={"ID":"df3c1bed-7286-448d-a3d0-b4e958f4fee1","Type":"ContainerDied","Data":"33b55177fccbfb78b0aa284304e476650a6f24ba9a7df9e190de2d944cdb37d0"}
Sep 17 21:39:50.101490 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.101188    2577 scope.go:117] "RemoveContainer" containerID="cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a"
Sep 17 21:39:50.110037 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.110016    2577 scope.go:117] "RemoveContainer" containerID="67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a"
Sep 17 21:39:50.117809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.117789    2577 scope.go:117] "RemoveContainer" containerID="d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de"
Sep 17 21:39:50.124497 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.124474    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"]
Sep 17 21:39:50.125954 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.125941    2577 scope.go:117] "RemoveContainer" containerID="cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a"
Sep 17 21:39:50.126210 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:39:50.126194    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a\": container with ID starting with cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a not found: ID does not exist" containerID="cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a"
Sep 17 21:39:50.126305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.126219    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a"} err="failed to get container status \"cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a\": rpc error: code = NotFound desc = could not find container \"cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a\": container with ID starting with cdb1f5da48cf732572884c86f4adcce117b14e13124b4f49e90441106cb21c6a not found: ID does not exist"
Sep 17 21:39:50.126305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.126252    2577 scope.go:117] "RemoveContainer" containerID="67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a"
Sep 17 21:39:50.126499 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:39:50.126479    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a\": container with ID starting with 67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a not found: ID does not exist" containerID="67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a"
Sep 17 21:39:50.126547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.126513    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a"} err="failed to get container status \"67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a\": rpc error: code = NotFound desc = could not find container \"67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a\": container with ID starting with 67a0c3d4454e12e8911b22a0e2130ebeb2136429afe9aca149b91c354f402b4a not found: ID does not exist"
Sep 17 21:39:50.126547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.126531    2577 scope.go:117] "RemoveContainer" containerID="d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de"
Sep 17 21:39:50.126865 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:39:50.126796    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de\": container with ID starting with d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de not found: ID does not exist" containerID="d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de"
Sep 17 21:39:50.126937 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.126873    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de"} err="failed to get container status \"d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de\": rpc error: code = NotFound desc = could not find container \"d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de\": container with ID starting with d7784f426092a55121cdc3d1c7ed335ee03139b172648bd4bc86c44ee203b9de not found: ID does not exist"
Sep 17 21:39:50.129503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:50.129483    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67c8ffcff9-qbl4m"]
Sep 17 21:39:51.718473 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:51.718440    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" path="/var/lib/kubelet/pods/df3c1bed-7286-448d-a3d0-b4e958f4fee1/volumes"
Sep 17 21:39:56.354640 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:56.354598    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:39:56.355108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:39:56.354713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:40:06.354438 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:06.354394    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:40:16.354752 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:16.354663    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:40:18.700916 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:18.700889    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:40:18.849255 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:18.849200    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8rdq\" (UniqueName: \"kubernetes.io/projected/3336bacd-9167-477d-9e16-cf30be42b4d2-kube-api-access-h8rdq\") pod \"3336bacd-9167-477d-9e16-cf30be42b4d2\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") "
Sep 17 21:40:18.849437 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:18.849297    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3336bacd-9167-477d-9e16-cf30be42b4d2-tls-certs\") pod \"3336bacd-9167-477d-9e16-cf30be42b4d2\" (UID: \"3336bacd-9167-477d-9e16-cf30be42b4d2\") "
Sep 17 21:40:18.851424 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:18.851391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3336bacd-9167-477d-9e16-cf30be42b4d2-kube-api-access-h8rdq" (OuterVolumeSpecName: "kube-api-access-h8rdq") pod "3336bacd-9167-477d-9e16-cf30be42b4d2" (UID: "3336bacd-9167-477d-9e16-cf30be42b4d2"). InnerVolumeSpecName "kube-api-access-h8rdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:40:18.851571 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:18.851545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3336bacd-9167-477d-9e16-cf30be42b4d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3336bacd-9167-477d-9e16-cf30be42b4d2" (UID: "3336bacd-9167-477d-9e16-cf30be42b4d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:40:18.950019 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:18.949982    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3336bacd-9167-477d-9e16-cf30be42b4d2-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:40:18.950019 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:18.950013    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rdq\" (UniqueName: \"kubernetes.io/projected/3336bacd-9167-477d-9e16-cf30be42b4d2-kube-api-access-h8rdq\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:40:19.242842 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.242760    2577 generic.go:356] "Generic (PLEG): container finished" podID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerID="822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93" exitCode=0
Sep 17 21:40:19.242842 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.242815    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"
Sep 17 21:40:19.243058 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.242842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" event={"ID":"3336bacd-9167-477d-9e16-cf30be42b4d2","Type":"ContainerDied","Data":"822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93"}
Sep 17 21:40:19.243058 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.242905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr" event={"ID":"3336bacd-9167-477d-9e16-cf30be42b4d2","Type":"ContainerDied","Data":"e3da5bba8452081f924e796a41a7de7eac513be6a7e97dc849dbcc63d3312e23"}
Sep 17 21:40:19.243058 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.242937    2577 scope.go:117] "RemoveContainer" containerID="822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93"
Sep 17 21:40:19.251748 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.251726    2577 scope.go:117] "RemoveContainer" containerID="822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93"
Sep 17 21:40:19.252105 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:40:19.252043    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93\": container with ID starting with 822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93 not found: ID does not exist" containerID="822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93"
Sep 17 21:40:19.252105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.252080    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93"} err="failed to get container status \"822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93\": rpc error: code = NotFound desc = could not find container \"822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93\": container with ID starting with 822893c3df97a45c1a3fea406e6fdef9ba00aeeb246e6efa26a445c6071a4c93 not found: ID does not exist"
Sep 17 21:40:19.267718 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.267694    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"]
Sep 17 21:40:19.271295 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.271274    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48rsnpr"]
Sep 17 21:40:19.718158 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:19.718124    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" path="/var/lib/kubelet/pods/3336bacd-9167-477d-9e16-cf30be42b4d2/volumes"
Sep 17 21:40:23.728698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.728665    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"]
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.728955    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.728973    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.728991    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729000    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729011    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="llm-d-routing-sidecar"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729020    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="llm-d-routing-sidecar"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729030    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="storage-initializer"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729039    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="storage-initializer"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729057    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729066    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729145    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="llm-d-routing-sidecar"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729159    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77d1355-634a-4520-84ac-ebdf8d24cbde" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729169    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3c1bed-7286-448d-a3d0-b4e958f4fee1" containerName="main"
Sep 17 21:40:23.729194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.729178    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3336bacd-9167-477d-9e16-cf30be42b4d2" containerName="main"
Sep 17 21:40:23.733950 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.733933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.736625 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.736602    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 21:40:23.736963 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.736939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 21:40:23.737122 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.736959    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 21:40:23.737122 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.737002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-9lh4j\""
Sep 17 21:40:23.748877 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.748847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"]
Sep 17 21:40:23.888719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.888719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.888932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66xb\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-kube-api-access-p66xb\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.888932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.888932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.888932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.888932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.888932 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.889136 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.888941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990095 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.989998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990095 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990318 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990318 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66xb\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-kube-api-access-p66xb\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990556 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990556 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990696 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990762 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990762 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990839 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.990875 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.990861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.993037 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.993017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.993176 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.993158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.998891 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.998853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:23.999030 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:23.998894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66xb\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-kube-api-access-p66xb\") pod \"router-gateway-2-openshift-default-78888675c6-2p485\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:24.044972 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:24.044940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:24.323922 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:24.323847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"]
Sep 17 21:40:24.326177 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:40:24.326145    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7a31fb_1cb5_4acf_93ed_0b1f93f273f5.slice/crio-69421f7caf4aacb63dd12305b9b35af294c96b0e7cff97f7f8f63ecfb50756ee WatchSource:0}: Error finding container 69421f7caf4aacb63dd12305b9b35af294c96b0e7cff97f7f8f63ecfb50756ee: Status 404 returned error can't find the container with id 69421f7caf4aacb63dd12305b9b35af294c96b0e7cff97f7f8f63ecfb50756ee
Sep 17 21:40:24.328139 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:24.328105    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:40:24.328275 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:24.328172    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:40:24.328275 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:24.328205    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:40:25.275749 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:25.275708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" event={"ID":"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5","Type":"ContainerStarted","Data":"828c2545e11bbe66591a4b4c497e0368a341880308bc0484c918f532d8af3935"}
Sep 17 21:40:25.276135 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:25.275768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" event={"ID":"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5","Type":"ContainerStarted","Data":"69421f7caf4aacb63dd12305b9b35af294c96b0e7cff97f7f8f63ecfb50756ee"}
Sep 17 21:40:25.300583 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:25.300538    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" podStartSLOduration=2.300522485 podStartE2EDuration="2.300522485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:40:25.298843589 +0000 UTC m=+3064.118135553" watchObservedRunningTime="2026-09-17 21:40:25.300522485 +0000 UTC m=+3064.119814508"
Sep 17 21:40:26.045414 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:26.045384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:26.050268 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:26.050219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:26.280545 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:26.280511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:26.281475 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:26.281457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:35.325180 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:35.325146    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerID="828c2545e11bbe66591a4b4c497e0368a341880308bc0484c918f532d8af3935" exitCode=0
Sep 17 21:40:35.325767 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:35.325228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" event={"ID":"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5","Type":"ContainerDied","Data":"828c2545e11bbe66591a4b4c497e0368a341880308bc0484c918f532d8af3935"}
Sep 17 21:40:35.325767 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:35.325698    2577 scope.go:117] "RemoveContainer" containerID="828c2545e11bbe66591a4b4c497e0368a341880308bc0484c918f532d8af3935"
Sep 17 21:40:35.326664 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:35.326634    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:40:35.326719 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:35.326705    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:40:35.326754 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:35.326738    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 17 21:40:36.331934 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:36.331894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" event={"ID":"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5","Type":"ContainerStarted","Data":"555ce9f172d70da51bd464c93ad6ab4bf831fd6ef7b7ed333a807e9938949f44"}
Sep 17 21:40:37.046951 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:37.046911    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:37.047363 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:37.047335    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 17 21:40:38.046035 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:38.045986    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 17 21:40:39.045436 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:39.045392    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 17 21:40:40.046108 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:40.046059    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 17 21:40:41.046333 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:41.046289    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 17 21:40:41.281094 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:41.281066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:42.049767 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:42.049736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:42.050879 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:42.050845    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:40:43.756887 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.756853    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"]
Sep 17 21:40:43.760635 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.760604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.763433 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.763394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:40:43.763433 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.763424    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 21:40:43.763621 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.763456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 21:40:43.763621 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.763481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-bqnx9\""
Sep 17 21:40:43.763697 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.763659    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-k9qhp\""
Sep 17 21:40:43.770664 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.770638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-dshm\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.770784 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.770696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-home\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.770784 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.770728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-model-cache\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.770916 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.770793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.770916 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.770853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rgnl\" (UniqueName: \"kubernetes.io/projected/4885c353-9e0e-4b9d-8751-8d980f751815-kube-api-access-4rgnl\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.770916 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.770892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-tmp-dir\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.771074 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.771005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4885c353-9e0e-4b9d-8751-8d980f751815-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.771074 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.771052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4885c353-9e0e-4b9d-8751-8d980f751815-tls-certs\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.775508 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.775487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"]
Sep 17 21:40:43.872359 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.872555 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rgnl\" (UniqueName: \"kubernetes.io/projected/4885c353-9e0e-4b9d-8751-8d980f751815-kube-api-access-4rgnl\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.872555 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-tmp-dir\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.872555 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4885c353-9e0e-4b9d-8751-8d980f751815-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.872555 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4885c353-9e0e-4b9d-8751-8d980f751815-tls-certs\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.872555 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-dshm\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.872555 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-home\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.872555 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-model-cache\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.873000 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.873000 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-home\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.873000 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-tmp-dir\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.873000 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.872957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-model-cache\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.873226 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.873194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4885c353-9e0e-4b9d-8751-8d980f751815-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.874942 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.874925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-dshm\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.875112 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.875095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4885c353-9e0e-4b9d-8751-8d980f751815-tls-certs\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:43.880273 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:43.880228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rgnl\" (UniqueName: \"kubernetes.io/projected/4885c353-9e0e-4b9d-8751-8d980f751815-kube-api-access-4rgnl\") pod \"router-with-refs-pd-test-kserve-66dc9dd865-4prhq\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:44.073806 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:44.073718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:44.358957 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:44.358929    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"]
Sep 17 21:40:44.361461 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:40:44.361434    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4885c353_9e0e_4b9d_8751_8d980f751815.slice/crio-c6242185ff3fccfabf8f40c1bf2128ce3aaad67fb992f91618fd71b556d3358f WatchSource:0}: Error finding container c6242185ff3fccfabf8f40c1bf2128ce3aaad67fb992f91618fd71b556d3358f: Status 404 returned error can't find the container with id c6242185ff3fccfabf8f40c1bf2128ce3aaad67fb992f91618fd71b556d3358f
Sep 17 21:40:44.385550 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:44.385315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerStarted","Data":"c6242185ff3fccfabf8f40c1bf2128ce3aaad67fb992f91618fd71b556d3358f"}
Sep 17 21:40:45.392566 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:45.392535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerStarted","Data":"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"}
Sep 17 21:40:45.392969 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:45.392625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:46.400182 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:46.400147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerStarted","Data":"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"}
Sep 17 21:40:48.413316 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:48.413278    2577 generic.go:356] "Generic (PLEG): container finished" podID="4885c353-9e0e-4b9d-8751-8d980f751815" containerID="49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701" exitCode=0
Sep 17 21:40:48.413792 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:48.413353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerDied","Data":"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"}
Sep 17 21:40:49.421897 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:49.421862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerStarted","Data":"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"}
Sep 17 21:40:49.447987 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:49.447917    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podStartSLOduration=4.217488168 podStartE2EDuration="6.447895611s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.230407443s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:40:49.444956764 +0000 UTC m=+3088.264248732" watchObservedRunningTime="2026-09-17 21:40:49.447895611 +0000 UTC m=+3088.267187575"
Sep 17 21:40:54.074471 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:54.074425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:54.074997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:54.074503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:40:54.075911 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:54.075868    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:40:54.105548 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:40:54.105514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:41:04.074479 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:41:04.074426    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:41:14.074254 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:41:14.074198    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:41:24.074819 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:41:24.074768    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:41:34.074618 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:41:34.074568    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:41:44.075045 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:41:44.074999    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:41:54.074068 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:41:54.074021    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:42:04.074976 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:42:04.074927    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:42:14.074352 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:42:14.074296    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:42:24.075092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:42:24.075045    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:42:34.074924 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:42:34.074854    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 17 21:42:44.085430 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:42:44.085395    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:42:44.098066 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:42:44.098038    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:42:53.297194 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:42:53.297155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"]
Sep 17 21:43:08.325832 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:08.325791    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main" containerID="cri-o://35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096" gracePeriod=45
Sep 17 21:43:08.843380 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:08.843328    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="llm-d-routing-sidecar" containerID="cri-o://07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba" gracePeriod=45
Sep 17 21:43:09.092603 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.092578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:43:09.155396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.155360    2577 generic.go:356] "Generic (PLEG): container finished" podID="4885c353-9e0e-4b9d-8751-8d980f751815" containerID="35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096" exitCode=0
Sep 17 21:43:09.155396 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.155390    2577 generic.go:356] "Generic (PLEG): container finished" podID="4885c353-9e0e-4b9d-8751-8d980f751815" containerID="07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba" exitCode=0
Sep 17 21:43:09.155591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.155414    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"
Sep 17 21:43:09.155591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.155430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerDied","Data":"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"}
Sep 17 21:43:09.155591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.155479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerDied","Data":"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"}
Sep 17 21:43:09.155591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.155498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq" event={"ID":"4885c353-9e0e-4b9d-8751-8d980f751815","Type":"ContainerDied","Data":"c6242185ff3fccfabf8f40c1bf2128ce3aaad67fb992f91618fd71b556d3358f"}
Sep 17 21:43:09.155591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.155518    2577 scope.go:117] "RemoveContainer" containerID="35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"
Sep 17 21:43:09.157763 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.157743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-kserve-provision-location\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.157840 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.157802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4885c353-9e0e-4b9d-8751-8d980f751815-tls-certs\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.157840 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.157829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4885c353-9e0e-4b9d-8751-8d980f751815-cabundle-cert\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.157923 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.157864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-tmp-dir\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.157923 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.157895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-model-cache\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.158020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.157926    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rgnl\" (UniqueName: \"kubernetes.io/projected/4885c353-9e0e-4b9d-8751-8d980f751815-kube-api-access-4rgnl\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.158020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.157978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-home\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.158020 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.158006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-dshm\") pod \"4885c353-9e0e-4b9d-8751-8d980f751815\" (UID: \"4885c353-9e0e-4b9d-8751-8d980f751815\") "
Sep 17 21:43:09.158286 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.158225    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-model-cache" (OuterVolumeSpecName: "model-cache") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.158286 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.158275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4885c353-9e0e-4b9d-8751-8d980f751815-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:43:09.159105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.159052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-home" (OuterVolumeSpecName: "home") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.160727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.160683    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-dshm" (OuterVolumeSpecName: "dshm") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.160829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.160728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4885c353-9e0e-4b9d-8751-8d980f751815-kube-api-access-4rgnl" (OuterVolumeSpecName: "kube-api-access-4rgnl") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "kube-api-access-4rgnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:43:09.160829 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.160778    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4885c353-9e0e-4b9d-8751-8d980f751815-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:43:09.166700 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.166680    2577 scope.go:117] "RemoveContainer" containerID="49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"
Sep 17 21:43:09.173423 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.173400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.176191 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.176166    2577 scope.go:117] "RemoveContainer" containerID="07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"
Sep 17 21:43:09.183830 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.183815    2577 scope.go:117] "RemoveContainer" containerID="35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"
Sep 17 21:43:09.184120 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:43:09.184100    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096\": container with ID starting with 35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096 not found: ID does not exist" containerID="35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"
Sep 17 21:43:09.184178 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184129    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"} err="failed to get container status \"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096\": rpc error: code = NotFound desc = could not find container \"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096\": container with ID starting with 35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096 not found: ID does not exist"
Sep 17 21:43:09.184178 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184147    2577 scope.go:117] "RemoveContainer" containerID="49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"
Sep 17 21:43:09.184423 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:43:09.184408    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701\": container with ID starting with 49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701 not found: ID does not exist" containerID="49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"
Sep 17 21:43:09.184483 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184426    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"} err="failed to get container status \"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701\": rpc error: code = NotFound desc = could not find container \"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701\": container with ID starting with 49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701 not found: ID does not exist"
Sep 17 21:43:09.184483 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184450    2577 scope.go:117] "RemoveContainer" containerID="07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"
Sep 17 21:43:09.184719 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:43:09.184700    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba\": container with ID starting with 07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba not found: ID does not exist" containerID="07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"
Sep 17 21:43:09.184774 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184731    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"} err="failed to get container status \"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba\": rpc error: code = NotFound desc = could not find container \"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba\": container with ID starting with 07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba not found: ID does not exist"
Sep 17 21:43:09.184774 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184746    2577 scope.go:117] "RemoveContainer" containerID="35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"
Sep 17 21:43:09.184984 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184969    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096"} err="failed to get container status \"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096\": rpc error: code = NotFound desc = could not find container \"35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096\": container with ID starting with 35d7fd5c28d660d96fb7d398fafa59933fa9f689c8996994fb509f72835e1096 not found: ID does not exist"
Sep 17 21:43:09.185028 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.184986    2577 scope.go:117] "RemoveContainer" containerID="49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"
Sep 17 21:43:09.185281 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.185218    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701"} err="failed to get container status \"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701\": rpc error: code = NotFound desc = could not find container \"49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701\": container with ID starting with 49dc98d627c6fa03591df58ec9cf8affa44a15ebc65415b220ee59d66656d701 not found: ID does not exist"
Sep 17 21:43:09.185281 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.185262    2577 scope.go:117] "RemoveContainer" containerID="07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"
Sep 17 21:43:09.185503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.185487    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba"} err="failed to get container status \"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba\": rpc error: code = NotFound desc = could not find container \"07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba\": container with ID starting with 07c37acd87d0265295e670591f4136599eb0808944c6c0919449d072ff7265ba not found: ID does not exist"
Sep 17 21:43:09.186930 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.186909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4885c353-9e0e-4b9d-8751-8d980f751815" (UID: "4885c353-9e0e-4b9d-8751-8d980f751815"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.259123 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259088    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-kserve-provision-location\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.259123 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259123    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4885c353-9e0e-4b9d-8751-8d980f751815-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.259123 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259133    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4885c353-9e0e-4b9d-8751-8d980f751815-cabundle-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.259314 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259142    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.259314 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259150    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.259314 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259157    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rgnl\" (UniqueName: \"kubernetes.io/projected/4885c353-9e0e-4b9d-8751-8d980f751815-kube-api-access-4rgnl\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.259314 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259165    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.259314 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.259172    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4885c353-9e0e-4b9d-8751-8d980f751815-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.478980 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.478939    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"]
Sep 17 21:43:09.483957 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.483932    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-66dc9dd865-4prhq"]
Sep 17 21:43:09.717430 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:09.717339    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" path="/var/lib/kubelet/pods/4885c353-9e0e-4b9d-8751-8d980f751815/volumes"
Sep 17 21:43:48.253091 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:48.253054    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"]
Sep 17 21:43:48.253693 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:48.253260    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy" containerID="cri-o://555ce9f172d70da51bd464c93ad6ab4bf831fd6ef7b7ed333a807e9938949f44" gracePeriod=30
Sep 17 21:43:53.375384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.375352    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerID="555ce9f172d70da51bd464c93ad6ab4bf831fd6ef7b7ed333a807e9938949f44" exitCode=0
Sep 17 21:43:53.375760 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.375439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" event={"ID":"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5","Type":"ContainerDied","Data":"555ce9f172d70da51bd464c93ad6ab4bf831fd6ef7b7ed333a807e9938949f44"}
Sep 17 21:43:53.375760 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.375518    2577 scope.go:117] "RemoveContainer" containerID="828c2545e11bbe66591a4b4c497e0368a341880308bc0484c918f532d8af3935"
Sep 17 21:43:53.526592 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.526501    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:43:53.626651 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.626611    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-token\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.626857 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.626680    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-envoy\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.626857 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.626793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-credential-socket\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.626990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.626853    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p66xb\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-kube-api-access-p66xb\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.626990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.626892    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istiod-ca-cert\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.626990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.626944    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-socket\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.626990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.626975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-data\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.627188 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.627034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-certs\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.627188 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.627103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-podinfo\") pod \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\" (UID: \"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5\") "
Sep 17 21:43:53.628034 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.627815    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:43:53.628379 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.628307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:53.628379 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.628328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:53.628547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.628374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:53.628547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.628433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:53.629514 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.629486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:53.629684 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.629666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:43:53.630071 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.630042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 21:43:53.630584 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.630555    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-kube-api-access-p66xb" (OuterVolumeSpecName: "kube-api-access-p66xb") pod "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" (UID: "cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5"). InnerVolumeSpecName "kube-api-access-p66xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:43:53.728822 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728785    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.728822 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728812    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-podinfo\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.728822 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728820    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-token\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.728822 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728828    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-envoy\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.729050 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728836    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-credential-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.729050 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728844    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66xb\" (UniqueName: \"kubernetes.io/projected/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-kube-api-access-p66xb\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.729050 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728854    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istiod-ca-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.729050 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728861    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-workload-socket\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:53.729050 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:53.728868    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5-istio-data\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:43:54.381706 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:54.381669    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"
Sep 17 21:43:54.381706 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:54.381680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485" event={"ID":"cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5","Type":"ContainerDied","Data":"69421f7caf4aacb63dd12305b9b35af294c96b0e7cff97f7f8f63ecfb50756ee"}
Sep 17 21:43:54.382254 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:54.381740    2577 scope.go:117] "RemoveContainer" containerID="555ce9f172d70da51bd464c93ad6ab4bf831fd6ef7b7ed333a807e9938949f44"
Sep 17 21:43:54.402081 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:54.402053    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"]
Sep 17 21:43:54.405792 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:54.405770    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-2p485"]
Sep 17 21:43:55.717613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:43:55.717579    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" path="/var/lib/kubelet/pods/cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5/volumes"
Sep 17 21:44:30.057547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.057506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"]
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.057922    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="llm-d-routing-sidecar"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.057941    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="llm-d-routing-sidecar"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.057962    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="storage-initializer"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.057970    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="storage-initializer"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.057988    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.057997    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058010    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058019    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058036    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy"
Sep 17 21:44:30.058077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058047    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy"
Sep 17 21:44:30.058614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058114    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy"
Sep 17 21:44:30.058614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058126    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="llm-d-routing-sidecar"
Sep 17 21:44:30.058614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058138    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7a31fb-1cb5-4acf-93ed-0b1f93f273f5" containerName="istio-proxy"
Sep 17 21:44:30.058614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.058147    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4885c353-9e0e-4b9d-8751-8d980f751815" containerName="main"
Sep 17 21:44:30.062371 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.062350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:30.065318 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.065296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-cwd5h\""
Sep 17 21:44:30.065420 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.065335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 21:44:30.065828 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.065814    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 21:44:30.065878 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.065827    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 21:44:30.070393 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.070373    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"]
Sep 17 21:44:30.196149 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.196117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfss\" (UniqueName: \"kubernetes.io/projected/44a6976a-d182-4765-9a96-a7187c57790b-kube-api-access-7nfss\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:30.296690 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.296645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfss\" (UniqueName: \"kubernetes.io/projected/44a6976a-d182-4765-9a96-a7187c57790b-kube-api-access-7nfss\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:30.305161 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:30.305141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfss\" (UniqueName: \"kubernetes.io/projected/44a6976a-d182-4765-9a96-a7187c57790b-kube-api-access-7nfss\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:38.657295 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:38.657255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:38.663428 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:38.663403    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:38.757695 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:38.757654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:38.757846 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:38.757756    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:38.760671 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:38.760653    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") DevicePath \"csi-9bec6de95015868d567cebddec3766a2f64f5c6368e309e890cb98153c6cf717\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:38.981775 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:38.981730    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/12a50bbb76c4de2cb27bb17a5d083445eac89e4a1524cdf9ead17a743718c4ae/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:38.996212 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:38.996184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"e2e-pvc-lora-download-z5bd6\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:39.072297 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:39.072263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:39.345684 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:39.345657    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"]
Sep 17 21:44:39.347908 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:44:39.347880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44a6976a_d182_4765_9a96_a7187c57790b.slice/crio-2d9f7064831a6bbb64fb5fd18fbdba840ac8b069ce92fa97c45664850c9c909c WatchSource:0}: Error finding container 2d9f7064831a6bbb64fb5fd18fbdba840ac8b069ce92fa97c45664850c9c909c: Status 404 returned error can't find the container with id 2d9f7064831a6bbb64fb5fd18fbdba840ac8b069ce92fa97c45664850c9c909c
Sep 17 21:44:39.605536 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:39.605444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6" event={"ID":"44a6976a-d182-4765-9a96-a7187c57790b","Type":"ContainerStarted","Data":"06efff36c9dc403747bc6b9b25ce32414bd4a9e923957e54633b778563d93ad5"}
Sep 17 21:44:39.605536 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:39.605492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6" event={"ID":"44a6976a-d182-4765-9a96-a7187c57790b","Type":"ContainerStarted","Data":"2d9f7064831a6bbb64fb5fd18fbdba840ac8b069ce92fa97c45664850c9c909c"}
Sep 17 21:44:39.621962 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:39.621908    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6" podStartSLOduration=12.621892078 podStartE2EDuration="12.621892078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:44:39.620695732 +0000 UTC m=+3318.439987694" watchObservedRunningTime="2026-09-17 21:44:39.621892078 +0000 UTC m=+3318.441184040"
Sep 17 21:44:41.616814 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:41.616779    2577 generic.go:356] "Generic (PLEG): container finished" podID="44a6976a-d182-4765-9a96-a7187c57790b" containerID="06efff36c9dc403747bc6b9b25ce32414bd4a9e923957e54633b778563d93ad5" exitCode=0
Sep 17 21:44:41.617308 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:41.616859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6" event={"ID":"44a6976a-d182-4765-9a96-a7187c57790b","Type":"ContainerDied","Data":"06efff36c9dc403747bc6b9b25ce32414bd4a9e923957e54633b778563d93ad5"}
Sep 17 21:44:42.759702 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:42.759677    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:42.891105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:42.891017    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nfss\" (UniqueName: \"kubernetes.io/projected/44a6976a-d182-4765-9a96-a7187c57790b-kube-api-access-7nfss\") pod \"44a6976a-d182-4765-9a96-a7187c57790b\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") "
Sep 17 21:44:42.891306 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:42.891126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"44a6976a-d182-4765-9a96-a7187c57790b\" (UID: \"44a6976a-d182-4765-9a96-a7187c57790b\") "
Sep 17 21:44:42.893444 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:42.893414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a6976a-d182-4765-9a96-a7187c57790b-kube-api-access-7nfss" (OuterVolumeSpecName: "kube-api-access-7nfss") pod "44a6976a-d182-4765-9a96-a7187c57790b" (UID: "44a6976a-d182-4765-9a96-a7187c57790b"). InnerVolumeSpecName "kube-api-access-7nfss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:44:42.895715 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:42.895689    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c" (OuterVolumeSpecName: "model-storage") pod "44a6976a-d182-4765-9a96-a7187c57790b" (UID: "44a6976a-d182-4765-9a96-a7187c57790b"). InnerVolumeSpecName "pvc-086a28a0-d1e1-4c31-a785-153afa6659c4". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 21:44:42.991925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:42.991890    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") on node \"ip-10-0-134-35.ec2.internal\" "
Sep 17 21:44:42.991925 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:42.991920    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfss\" (UniqueName: \"kubernetes.io/projected/44a6976a-d182-4765-9a96-a7187c57790b-kube-api-access-7nfss\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:44:43.633379 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:43.633336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6" event={"ID":"44a6976a-d182-4765-9a96-a7187c57790b","Type":"ContainerDied","Data":"2d9f7064831a6bbb64fb5fd18fbdba840ac8b069ce92fa97c45664850c9c909c"}
Sep 17 21:44:43.633379 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:43.633355    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"
Sep 17 21:44:43.633379 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:43.633374    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d9f7064831a6bbb64fb5fd18fbdba840ac8b069ce92fa97c45664850c9c909c"
Sep 17 21:44:43.972253 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:43.972215    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-086a28a0-d1e1-4c31-a785-153afa6659c4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c") on node "ip-10-0-134-35.ec2.internal"
Sep 17 21:44:44.001521 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:44.001494    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"csi-9bec6de95015868d567cebddec3766a2f64f5c6368e309e890cb98153c6cf717\""
Sep 17 21:44:53.014561 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.014519    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"]
Sep 17 21:44:53.014951 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.014882    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a6976a-d182-4765-9a96-a7187c57790b" containerName="storage-initializer"
Sep 17 21:44:53.014951 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.014904    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a6976a-d182-4765-9a96-a7187c57790b" containerName="storage-initializer"
Sep 17 21:44:53.015027 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.014966    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a6976a-d182-4765-9a96-a7187c57790b" containerName="storage-initializer"
Sep 17 21:44:53.018126 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.018108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.020867 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.020830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 21:44:53.020867 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.020850    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:44:53.021059 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.020926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 21:44:53.021059 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.020960    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 21:44:53.021251 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.021225    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 21:44:53.021301 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.021273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-cwd5h\""
Sep 17 21:44:53.033065 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.033044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"]
Sep 17 21:44:53.066913 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.066871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.066913 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.066916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14977da7-9f94-4864-88ea-15bb0994be12-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.067115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.066955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss54h\" (UniqueName: \"kubernetes.io/projected/14977da7-9f94-4864-88ea-15bb0994be12-kube-api-access-ss54h\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.067115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.066974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.067115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.066993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-dshm\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.067115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.067006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.067115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.067083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14977da7-9f94-4864-88ea-15bb0994be12-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.067115 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.067101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-home\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.167615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14977da7-9f94-4864-88ea-15bb0994be12-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.167771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss54h\" (UniqueName: \"kubernetes.io/projected/14977da7-9f94-4864-88ea-15bb0994be12-kube-api-access-ss54h\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.167771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.167771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-dshm\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.167771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.167771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14977da7-9f94-4864-88ea-15bb0994be12-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.167771 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-home\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.168044 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.167781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.168160 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.168127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.168319 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.168175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.168319 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.168224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.168319 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.168301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-home\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.168709 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.168647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14977da7-9f94-4864-88ea-15bb0994be12-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.170215 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.170192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-dshm\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.170383 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.170367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14977da7-9f94-4864-88ea-15bb0994be12-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:53.176285 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:53.176263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss54h\" (UniqueName: \"kubernetes.io/projected/14977da7-9f94-4864-88ea-15bb0994be12-kube-api-access-ss54h\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.114456 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.114415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.119563 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.119533    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.215300 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.215256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.215468 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.215320    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.218219 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.218198    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") DevicePath \"csi-9bec6de95015868d567cebddec3766a2f64f5c6368e309e890cb98153c6cf717\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.309080 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.309047    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/12a50bbb76c4de2cb27bb17a5d083445eac89e4a1524cdf9ead17a743718c4ae/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.328287 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.328223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.628701 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.628665    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:44:59.912199 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:44:59.912170    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"]
Sep 17 21:44:59.913990 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:44:59.913963    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14977da7_9f94_4864_88ea_15bb0994be12.slice/crio-ee3d4e0a13941a4fcb3523ce6e3e749b7910b5db12b8e285d7bb8e93aef282a7 WatchSource:0}: Error finding container ee3d4e0a13941a4fcb3523ce6e3e749b7910b5db12b8e285d7bb8e93aef282a7: Status 404 returned error can't find the container with id ee3d4e0a13941a4fcb3523ce6e3e749b7910b5db12b8e285d7bb8e93aef282a7
Sep 17 21:45:00.717707 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:00.717670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" event={"ID":"14977da7-9f94-4864-88ea-15bb0994be12","Type":"ContainerStarted","Data":"fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18"}
Sep 17 21:45:00.718158 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:00.717723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" event={"ID":"14977da7-9f94-4864-88ea-15bb0994be12","Type":"ContainerStarted","Data":"ee3d4e0a13941a4fcb3523ce6e3e749b7910b5db12b8e285d7bb8e93aef282a7"}
Sep 17 21:45:02.727587 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:02.727552    2577 generic.go:356] "Generic (PLEG): container finished" podID="14977da7-9f94-4864-88ea-15bb0994be12" containerID="fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18" exitCode=0
Sep 17 21:45:02.727986 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:02.727624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" event={"ID":"14977da7-9f94-4864-88ea-15bb0994be12","Type":"ContainerDied","Data":"fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18"}
Sep 17 21:45:03.735556 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:03.735519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" event={"ID":"14977da7-9f94-4864-88ea-15bb0994be12","Type":"ContainerStarted","Data":"3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4"}
Sep 17 21:45:03.760853 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:03.760803    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podStartSLOduration=9.636329074 podStartE2EDuration="11.760784746s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.124455672s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:45:03.759491755 +0000 UTC m=+3342.578783717" watchObservedRunningTime="2026-09-17 21:45:03.760784746 +0000 UTC m=+3342.580076708"
Sep 17 21:45:04.629502 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:04.629460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:45:09.629005 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:09.628963    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:45:09.630428 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:09.630399    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 21:45:19.630058 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:19.630007    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 21:45:29.629705 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:29.629655    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 21:45:39.629145 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:39.629100    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 21:45:49.629856 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:49.629810    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 21:45:59.629105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:45:59.629057    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 21:46:09.629103 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:46:09.629061    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 21:46:19.639720 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:46:19.639629    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:46:19.647505 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:46:19.647473    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:47:09.651096 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:09.649355    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v_14977da7-9f94-4864-88ea-15bb0994be12/storage-initializer/0.log"
Sep 17 21:47:09.659698 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:09.659673    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v_14977da7-9f94-4864-88ea-15bb0994be12/main/0.log"
Sep 17 21:47:10.957289 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:10.957188    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"]
Sep 17 21:47:25.980015 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:25.979966    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main" containerID="cri-o://3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4" gracePeriod=45
Sep 17 21:47:26.639975 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.639952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:47:26.734939 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.734838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-dshm\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.734939 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.734873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-home\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.734939 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.734900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-kserve-provision-location\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.735221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.734974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14977da7-9f94-4864-88ea-15bb0994be12-tls-certs\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.735221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss54h\" (UniqueName: \"kubernetes.io/projected/14977da7-9f94-4864-88ea-15bb0994be12-kube-api-access-ss54h\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.735221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14977da7-9f94-4864-88ea-15bb0994be12-cabundle-cert\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.735221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.735221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-tmp-dir\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.735221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-model-cache\") pod \"14977da7-9f94-4864-88ea-15bb0994be12\" (UID: \"14977da7-9f94-4864-88ea-15bb0994be12\") "
Sep 17 21:47:26.735570 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-home" (OuterVolumeSpecName: "home") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:47:26.735570 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735476    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14977da7-9f94-4864-88ea-15bb0994be12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:47:26.735931 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.735904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-model-cache" (OuterVolumeSpecName: "model-cache") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:47:26.737368 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.737338    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14977da7-9f94-4864-88ea-15bb0994be12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:47:26.737368 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.737347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-dshm" (OuterVolumeSpecName: "dshm") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:47:26.737986 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.737956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14977da7-9f94-4864-88ea-15bb0994be12-kube-api-access-ss54h" (OuterVolumeSpecName: "kube-api-access-ss54h") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "kube-api-access-ss54h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:47:26.740567 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.740546    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "pvc-086a28a0-d1e1-4c31-a785-153afa6659c4". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 21:47:26.753772 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.753747    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:47:26.763978 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.763955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14977da7-9f94-4864-88ea-15bb0994be12" (UID: "14977da7-9f94-4864-88ea-15bb0994be12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:47:26.836517 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836484    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14977da7-9f94-4864-88ea-15bb0994be12-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.836517 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836512    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss54h\" (UniqueName: \"kubernetes.io/projected/14977da7-9f94-4864-88ea-15bb0994be12-kube-api-access-ss54h\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.836517 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836523    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14977da7-9f94-4864-88ea-15bb0994be12-cabundle-cert\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.836727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836547    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") on node \"ip-10-0-134-35.ec2.internal\" "
Sep 17 21:47:26.836727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836556    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.836727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836563    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.836727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836571    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.836727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836580    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.836727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.836587    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14977da7-9f94-4864-88ea-15bb0994be12-kserve-provision-location\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:26.847046 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.847026    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-086a28a0-d1e1-4c31-a785-153afa6659c4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c") on node "ip-10-0-134-35.ec2.internal"
Sep 17 21:47:26.937599 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:26.937569    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-086a28a0-d1e1-4c31-a785-153afa6659c4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b960408bf90c878c\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"csi-9bec6de95015868d567cebddec3766a2f64f5c6368e309e890cb98153c6cf717\""
Sep 17 21:47:27.464727 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.464687    2577 generic.go:356] "Generic (PLEG): container finished" podID="14977da7-9f94-4864-88ea-15bb0994be12" containerID="3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4" exitCode=0
Sep 17 21:47:27.465129 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.464739    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"
Sep 17 21:47:27.465129 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.464780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" event={"ID":"14977da7-9f94-4864-88ea-15bb0994be12","Type":"ContainerDied","Data":"3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4"}
Sep 17 21:47:27.465129 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.464838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v" event={"ID":"14977da7-9f94-4864-88ea-15bb0994be12","Type":"ContainerDied","Data":"ee3d4e0a13941a4fcb3523ce6e3e749b7910b5db12b8e285d7bb8e93aef282a7"}
Sep 17 21:47:27.465129 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.464865    2577 scope.go:117] "RemoveContainer" containerID="3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4"
Sep 17 21:47:27.473959 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.473940    2577 scope.go:117] "RemoveContainer" containerID="fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18"
Sep 17 21:47:27.482077 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.482060    2577 scope.go:117] "RemoveContainer" containerID="3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4"
Sep 17 21:47:27.482348 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:47:27.482323    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4\": container with ID starting with 3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4 not found: ID does not exist" containerID="3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4"
Sep 17 21:47:27.482410 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.482361    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4"} err="failed to get container status \"3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4\": rpc error: code = NotFound desc = could not find container \"3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4\": container with ID starting with 3a6aafac8bf8492f7ab843203bddda5bc04fe961fe0ce532c355a275cbb67ce4 not found: ID does not exist"
Sep 17 21:47:27.482410 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.482387    2577 scope.go:117] "RemoveContainer" containerID="fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18"
Sep 17 21:47:27.482704 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:47:27.482685    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18\": container with ID starting with fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18 not found: ID does not exist" containerID="fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18"
Sep 17 21:47:27.482763 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.482710    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18"} err="failed to get container status \"fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18\": rpc error: code = NotFound desc = could not find container \"fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18\": container with ID starting with fb28d574cb9c2f21dd6afc237938d0bfd6c166871fedd4c9271b161516e0de18 not found: ID does not exist"
Sep 17 21:47:27.498717 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.498696    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"]
Sep 17 21:47:27.508664 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.508637    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9d65cccb-gbh9v"]
Sep 17 21:47:27.720715 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:27.720637    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14977da7-9f94-4864-88ea-15bb0994be12" path="/var/lib/kubelet/pods/14977da7-9f94-4864-88ea-15bb0994be12/volumes"
Sep 17 21:47:34.332843 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.332807    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"]
Sep 17 21:47:34.333227 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.333106    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main"
Sep 17 21:47:34.333227 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.333117    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main"
Sep 17 21:47:34.333227 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.333130    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="storage-initializer"
Sep 17 21:47:34.333227 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.333136    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="storage-initializer"
Sep 17 21:47:34.333227 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.333183    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14977da7-9f94-4864-88ea-15bb0994be12" containerName="main"
Sep 17 21:47:34.337999 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.337981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:34.340502 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.340475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-45gww\""
Sep 17 21:47:34.340640 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.340504    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 21:47:34.340640 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.340475    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 21:47:34.341284 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.341267    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 21:47:34.347877 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.347852    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"]
Sep 17 21:47:34.497203 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.497172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfj4g\" (UniqueName: \"kubernetes.io/projected/4701a9eb-874b-4d21-9d07-8ea51e07fe15-kube-api-access-nfj4g\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:34.598619 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.598533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfj4g\" (UniqueName: \"kubernetes.io/projected/4701a9eb-874b-4d21-9d07-8ea51e07fe15-kube-api-access-nfj4g\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:34.607761 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:34.607738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfj4g\" (UniqueName: \"kubernetes.io/projected/4701a9eb-874b-4d21-9d07-8ea51e07fe15-kube-api-access-nfj4g\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:36.019027 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:36.018988    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"]
Sep 17 21:47:36.021446 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:36.021423    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-z5bd6"]
Sep 17 21:47:37.717351 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:37.717317    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a6976a-d182-4765-9a96-a7187c57790b" path="/var/lib/kubelet/pods/44a6976a-d182-4765-9a96-a7187c57790b/volumes"
Sep 17 21:47:42.263748 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.263713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:42.269505 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.269482    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:42.365016 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.364980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:42.365158 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.365111    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:42.368082 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.368064    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") DevicePath \"csi-4c6d371be02328330b542a23ca86b81c199d512c65389033f472dab8ad0015f3\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:42.577655 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.577618    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0a812c6ac6276fee7b3060498e32aeb0856aaa80b25585b600f523c8da4af4f4/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:42.590721 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.590693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"e2e-pvc-lora-download-c5zsl\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:42.747635 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:42.747602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:43.016071 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:43.016041    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"]
Sep 17 21:47:43.019215 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:47:43.019158    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4701a9eb_874b_4d21_9d07_8ea51e07fe15.slice/crio-a4075381d5e7a6f0f1d071b53697b97357b66a3a7dd3f89cbd1e5de2542bb79e WatchSource:0}: Error finding container a4075381d5e7a6f0f1d071b53697b97357b66a3a7dd3f89cbd1e5de2542bb79e: Status 404 returned error can't find the container with id a4075381d5e7a6f0f1d071b53697b97357b66a3a7dd3f89cbd1e5de2542bb79e
Sep 17 21:47:43.543186 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:43.543146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl" event={"ID":"4701a9eb-874b-4d21-9d07-8ea51e07fe15","Type":"ContainerStarted","Data":"425e5a54ac0b43fdeb56357f77815f1d43f40452124342fc25929708ad9510e3"}
Sep 17 21:47:43.543186 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:43.543188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl" event={"ID":"4701a9eb-874b-4d21-9d07-8ea51e07fe15","Type":"ContainerStarted","Data":"a4075381d5e7a6f0f1d071b53697b97357b66a3a7dd3f89cbd1e5de2542bb79e"}
Sep 17 21:47:43.561035 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:43.560978    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl" podStartSLOduration=12.56096118 podStartE2EDuration="12.56096118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:47:43.559401158 +0000 UTC m=+3502.378693150" watchObservedRunningTime="2026-09-17 21:47:43.56096118 +0000 UTC m=+3502.380253141"
Sep 17 21:47:45.553305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:45.553265    2577 generic.go:356] "Generic (PLEG): container finished" podID="4701a9eb-874b-4d21-9d07-8ea51e07fe15" containerID="425e5a54ac0b43fdeb56357f77815f1d43f40452124342fc25929708ad9510e3" exitCode=0
Sep 17 21:47:45.553688 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:45.553278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl" event={"ID":"4701a9eb-874b-4d21-9d07-8ea51e07fe15","Type":"ContainerDied","Data":"425e5a54ac0b43fdeb56357f77815f1d43f40452124342fc25929708ad9510e3"}
Sep 17 21:47:46.700331 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:46.700306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:46.802564 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:46.802522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfj4g\" (UniqueName: \"kubernetes.io/projected/4701a9eb-874b-4d21-9d07-8ea51e07fe15-kube-api-access-nfj4g\") pod \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") "
Sep 17 21:47:46.802757 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:46.802697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\" (UID: \"4701a9eb-874b-4d21-9d07-8ea51e07fe15\") "
Sep 17 21:47:46.805026 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:46.804999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4701a9eb-874b-4d21-9d07-8ea51e07fe15-kube-api-access-nfj4g" (OuterVolumeSpecName: "kube-api-access-nfj4g") pod "4701a9eb-874b-4d21-9d07-8ea51e07fe15" (UID: "4701a9eb-874b-4d21-9d07-8ea51e07fe15"). InnerVolumeSpecName "kube-api-access-nfj4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:47:46.805692 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:46.805674    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d" (OuterVolumeSpecName: "model-storage") pod "4701a9eb-874b-4d21-9d07-8ea51e07fe15" (UID: "4701a9eb-874b-4d21-9d07-8ea51e07fe15"). InnerVolumeSpecName "pvc-100f4501-9578-4e19-ba22-7c71733d3d39". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 21:47:46.903324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:46.903295    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") on node \"ip-10-0-134-35.ec2.internal\" "
Sep 17 21:47:46.903324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:46.903322    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfj4g\" (UniqueName: \"kubernetes.io/projected/4701a9eb-874b-4d21-9d07-8ea51e07fe15-kube-api-access-nfj4g\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:47:47.563755 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:47.563723    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"
Sep 17 21:47:47.563755 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:47.563736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl" event={"ID":"4701a9eb-874b-4d21-9d07-8ea51e07fe15","Type":"ContainerDied","Data":"a4075381d5e7a6f0f1d071b53697b97357b66a3a7dd3f89cbd1e5de2542bb79e"}
Sep 17 21:47:47.563976 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:47.563774    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4075381d5e7a6f0f1d071b53697b97357b66a3a7dd3f89cbd1e5de2542bb79e"
Sep 17 21:47:47.874233 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:47.874209    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-100f4501-9578-4e19-ba22-7c71733d3d39" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d") on node "ip-10-0-134-35.ec2.internal"
Sep 17 21:47:47.910564 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:47:47.910528    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"csi-4c6d371be02328330b542a23ca86b81c199d512c65389033f472dab8ad0015f3\""
Sep 17 21:48:02.927636 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.927598    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"]
Sep 17 21:48:02.928043 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.927889    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4701a9eb-874b-4d21-9d07-8ea51e07fe15" containerName="storage-initializer"
Sep 17 21:48:02.928043 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.927900    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4701a9eb-874b-4d21-9d07-8ea51e07fe15" containerName="storage-initializer"
Sep 17 21:48:02.928043 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.927973    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4701a9eb-874b-4d21-9d07-8ea51e07fe15" containerName="storage-initializer"
Sep 17 21:48:02.930972 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.930955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:02.934473 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.934445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 21:48:02.934889 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.934872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-45gww\""
Sep 17 21:48:02.935188 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.935171    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 21:48:02.935689 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.935674    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 21:48:02.945628 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:02.945606    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"]
Sep 17 21:48:03.036821 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.036787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.036997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.036828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.036997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.036863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgmq\" (UniqueName: \"kubernetes.io/projected/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-kube-api-access-blgmq\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.036997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.036958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.036997 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.036992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.037172 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.037011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137359 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137359 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgmq\" (UniqueName: \"kubernetes.io/projected/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-kube-api-access-blgmq\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137859 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137859 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.137859 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.137850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.140002 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.139979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.140227 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.140204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:03.146772 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:03.146747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgmq\" (UniqueName: \"kubernetes.io/projected/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-kube-api-access-blgmq\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.021047 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.021008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.027063 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.027034    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.122292 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.122257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.122471 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.122408    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.125348 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.125325    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") DevicePath \"csi-4c6d371be02328330b542a23ca86b81c199d512c65389033f472dab8ad0015f3\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.216423 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.216384    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0a812c6ac6276fee7b3060498e32aeb0856aaa80b25585b600f523c8da4af4f4/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.238040 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.238015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.440821 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.440729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:13.724666 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:13.724638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"]
Sep 17 21:48:13.726530 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:48:13.726504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f497ef9_a7f2_4b33_9066_dfb5b4c65d86.slice/crio-81eefe2d80d4b745254c7b3054c25db065aac6dfdbdd64c6e725e494ba224571 WatchSource:0}: Error finding container 81eefe2d80d4b745254c7b3054c25db065aac6dfdbdd64c6e725e494ba224571: Status 404 returned error can't find the container with id 81eefe2d80d4b745254c7b3054c25db065aac6dfdbdd64c6e725e494ba224571
Sep 17 21:48:14.702849 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:14.702813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" event={"ID":"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86","Type":"ContainerStarted","Data":"f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c"}
Sep 17 21:48:14.703277 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:14.702864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" event={"ID":"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86","Type":"ContainerStarted","Data":"81eefe2d80d4b745254c7b3054c25db065aac6dfdbdd64c6e725e494ba224571"}
Sep 17 21:48:14.728133 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:14.728060    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podStartSLOduration=12.728037854 podStartE2EDuration="12.728037854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:48:14.724321368 +0000 UTC m=+3533.543613330" watchObservedRunningTime="2026-09-17 21:48:14.728037854 +0000 UTC m=+3533.547329824"
Sep 17 21:48:15.440993 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:15.440959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:23.441839 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:23.441801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:48:23.443446 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:23.443415    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:48:33.442322 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:33.441850    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:48:43.441962 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:43.441918    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:48:53.441430 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:48:53.441381    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:49:03.441745 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:49:03.441692    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:49:13.441736 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:49:13.441648    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:49:23.441587 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:49:23.441528    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:49:33.451565 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:49:33.451531    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:49:33.459105 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:49:33.459078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:50:01.100093 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:01.100057    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp_0f497ef9-a7f2-4b33-9066-dfb5b4c65d86/main/0.log"
Sep 17 21:50:04.162552 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:04.162517    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"]
Sep 17 21:50:19.185455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.185387    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" containerID="cri-o://f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c" gracePeriod=45
Sep 17 21:50:19.453084 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.452988    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 21:50:19.873706 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.873680    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:50:19.945975 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.945934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-home\") pod \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") "
Sep 17 21:50:19.946154 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.945981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-model-cache\") pod \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") "
Sep 17 21:50:19.946154 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.946049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-dshm\") pod \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") "
Sep 17 21:50:19.946154 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.946076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tmp-dir\") pod \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") "
Sep 17 21:50:19.946384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.946198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") pod \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") "
Sep 17 21:50:19.946384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.946304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" (UID: "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:19.946488 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.946444    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-home" (OuterVolumeSpecName: "home") pod "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" (UID: "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:19.946611 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.946585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blgmq\" (UniqueName: \"kubernetes.io/projected/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-kube-api-access-blgmq\") pod \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") "
Sep 17 21:50:19.946774 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.946634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tls-certs\") pod \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\" (UID: \"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86\") "
Sep 17 21:50:19.947114 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.947082    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-model-cache\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:50:19.947114 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.947107    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-home\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:50:19.948591 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.948561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-dshm" (OuterVolumeSpecName: "dshm") pod "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" (UID: "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:19.948842 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.948817    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-kube-api-access-blgmq" (OuterVolumeSpecName: "kube-api-access-blgmq") pod "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" (UID: "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86"). InnerVolumeSpecName "kube-api-access-blgmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:50:19.949372 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.949352    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" (UID: "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:50:19.955032 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.955008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d" (OuterVolumeSpecName: "kserve-pvc-source") pod "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" (UID: "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86"). InnerVolumeSpecName "pvc-100f4501-9578-4e19-ba22-7c71733d3d39". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 21:50:19.960221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:19.960196    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" (UID: "0f497ef9-a7f2-4b33-9066-dfb5b4c65d86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:20.047561 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.047522    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgmq\" (UniqueName: \"kubernetes.io/projected/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-kube-api-access-blgmq\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.047561 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.047554    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.047561 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.047566    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-dshm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.047795 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.047576    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86-tmp-dir\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.047795 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.047605    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") on node \"ip-10-0-134-35.ec2.internal\" "
Sep 17 21:50:20.076480 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.076457    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-100f4501-9578-4e19-ba22-7c71733d3d39" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d") on node "ip-10-0-134-35.ec2.internal"
Sep 17 21:50:20.148810 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.148773    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-100f4501-9578-4e19-ba22-7c71733d3d39\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-037c3e81ee400e35d\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"csi-4c6d371be02328330b542a23ca86b81c199d512c65389033f472dab8ad0015f3\""
Sep 17 21:50:20.338455 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.338422    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerID="f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c" exitCode=0
Sep 17 21:50:20.338910 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.338471    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"
Sep 17 21:50:20.338910 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.338483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" event={"ID":"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86","Type":"ContainerDied","Data":"f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c"}
Sep 17 21:50:20.338910 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.338525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp" event={"ID":"0f497ef9-a7f2-4b33-9066-dfb5b4c65d86","Type":"ContainerDied","Data":"81eefe2d80d4b745254c7b3054c25db065aac6dfdbdd64c6e725e494ba224571"}
Sep 17 21:50:20.338910 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.338548    2577 scope.go:117] "RemoveContainer" containerID="f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c"
Sep 17 21:50:20.348737 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.348716    2577 scope.go:117] "RemoveContainer" containerID="f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c"
Sep 17 21:50:20.349008 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:50:20.348990    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c\": container with ID starting with f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c not found: ID does not exist" containerID="f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c"
Sep 17 21:50:20.349057 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.349015    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c"} err="failed to get container status \"f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c\": rpc error: code = NotFound desc = could not find container \"f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c\": container with ID starting with f4fcadfff5093817cb3a1fad9b6911eb4684fc8b7539c975411c74b20f83359c not found: ID does not exist"
Sep 17 21:50:20.365113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.365088    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"]
Sep 17 21:50:20.371963 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:20.371938    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-9hrcp"]
Sep 17 21:50:21.717707 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:21.717673    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" path="/var/lib/kubelet/pods/0f497ef9-a7f2-4b33-9066-dfb5b4c65d86/volumes"
Sep 17 21:50:27.234621 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:27.234587    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"]
Sep 17 21:50:27.238992 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:27.238965    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-c5zsl"]
Sep 17 21:50:27.717467 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:27.717431    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4701a9eb-874b-4d21-9d07-8ea51e07fe15" path="/var/lib/kubelet/pods/4701a9eb-874b-4d21-9d07-8ea51e07fe15/volumes"
Sep 17 21:50:34.131348 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.131309    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"]
Sep 17 21:50:34.131855 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.131788    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main"
Sep 17 21:50:34.131855 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.131809    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main"
Sep 17 21:50:34.131973 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.131872    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f497ef9-a7f2-4b33-9066-dfb5b4c65d86" containerName="main"
Sep 17 21:50:34.136439 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.136417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.139514 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.139492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-pn4lg\""
Sep 17 21:50:34.140210 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.140188    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 21:50:34.140356 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.140222    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-5jnsh\""
Sep 17 21:50:34.140459 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.140438    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 21:50:34.140689 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.140670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 21:50:34.149403 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.149378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"]
Sep 17 21:50:34.152281 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.152224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fdedc-ac58-4831-b820-72fa3e20a902-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.152380 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.152310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vsk\" (UniqueName: \"kubernetes.io/projected/0d3fdedc-ac58-4831-b820-72fa3e20a902-kube-api-access-k5vsk\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.253345 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.253304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vsk\" (UniqueName: \"kubernetes.io/projected/0d3fdedc-ac58-4831-b820-72fa3e20a902-kube-api-access-k5vsk\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.253529 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.253369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fdedc-ac58-4831-b820-72fa3e20a902-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.255981 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.255947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fdedc-ac58-4831-b820-72fa3e20a902-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.261978 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.261934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vsk\" (UniqueName: \"kubernetes.io/projected/0d3fdedc-ac58-4831-b820-72fa3e20a902-kube-api-access-k5vsk\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.447356 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.447265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:34.722509 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:34.722481    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"]
Sep 17 21:50:34.724792 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:50:34.724760    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3fdedc_ac58_4831_b820_72fa3e20a902.slice/crio-d772f740b284f65ac3fd8394eed36db9dc46ad335a89923a6931272a6d9bfdcc WatchSource:0}: Error finding container d772f740b284f65ac3fd8394eed36db9dc46ad335a89923a6931272a6d9bfdcc: Status 404 returned error can't find the container with id d772f740b284f65ac3fd8394eed36db9dc46ad335a89923a6931272a6d9bfdcc
Sep 17 21:50:35.412373 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:35.412339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" event={"ID":"0d3fdedc-ac58-4831-b820-72fa3e20a902","Type":"ContainerStarted","Data":"9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b"}
Sep 17 21:50:35.412783 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:35.412394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" event={"ID":"0d3fdedc-ac58-4831-b820-72fa3e20a902","Type":"ContainerStarted","Data":"d772f740b284f65ac3fd8394eed36db9dc46ad335a89923a6931272a6d9bfdcc"}
Sep 17 21:50:35.412783 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:35.412465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:50:35.432121 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:50:35.432074    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" podStartSLOduration=1.432059103 podStartE2EDuration="1.432059103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:50:35.429178465 +0000 UTC m=+3674.248470438" watchObservedRunningTime="2026-09-17 21:50:35.432059103 +0000 UTC m=+3674.251351064"
Sep 17 21:51:06.420293 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:06.420226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:51:15.924587 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.924553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"]
Sep 17 21:51:15.927226 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.927208    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:15.930524 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.930499    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 21:51:15.930757 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.930736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 21:51:15.930823 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.930784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-5956t\""
Sep 17 21:51:15.930881 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.930844    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-989ln\""
Sep 17 21:51:15.930952 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.930909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 21:51:15.944005 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:15.943983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"]
Sep 17 21:51:16.087937 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.087894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.088121 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.087962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhvv\" (UniqueName: \"kubernetes.io/projected/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-kube-api-access-rnhvv\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.188607 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.188507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.188776 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.188579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhvv\" (UniqueName: \"kubernetes.io/projected/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-kube-api-access-rnhvv\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.191481 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.191451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.198081 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.198056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhvv\" (UniqueName: \"kubernetes.io/projected/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-kube-api-access-rnhvv\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.238087 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.238051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.519011 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.518983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"]
Sep 17 21:51:16.520434 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:51:16.520409    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb448b77a_50a2_47c3_898f_3cd66f6b6f9e.slice/crio-6053aec4585e636c4c64b26c2ed24705e30e152e66eb05fe96c4c2693135397f WatchSource:0}: Error finding container 6053aec4585e636c4c64b26c2ed24705e30e152e66eb05fe96c4c2693135397f: Status 404 returned error can't find the container with id 6053aec4585e636c4c64b26c2ed24705e30e152e66eb05fe96c4c2693135397f
Sep 17 21:51:16.608193 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.608151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" event={"ID":"b448b77a-50a2-47c3-898f-3cd66f6b6f9e","Type":"ContainerStarted","Data":"e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f"}
Sep 17 21:51:16.608403 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.608213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" event={"ID":"b448b77a-50a2-47c3-898f-3cd66f6b6f9e","Type":"ContainerStarted","Data":"6053aec4585e636c4c64b26c2ed24705e30e152e66eb05fe96c4c2693135397f"}
Sep 17 21:51:16.608403 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.608397    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:16.627528 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:16.627482    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" podStartSLOduration=1.6274672350000001 podStartE2EDuration="1.627467235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:51:16.627122853 +0000 UTC m=+3715.446414815" watchObservedRunningTime="2026-09-17 21:51:16.627467235 +0000 UTC m=+3715.446759197"
Sep 17 21:51:22.220675 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:22.220640    2577 scope.go:117] "RemoveContainer" containerID="06efff36c9dc403747bc6b9b25ce32414bd4a9e923957e54633b778563d93ad5"
Sep 17 21:51:47.614736 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:47.614705    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:51:49.034582 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:49.034544    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"]
Sep 17 21:51:57.614763 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:51:57.614716    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:52:04.055535 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:04.055486    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main" containerID="cri-o://e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f" gracePeriod=45
Sep 17 21:52:07.614780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:07.614741    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:52:17.614687 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:17.614601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:52:17.615282 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:17.614706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:52:27.614984 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:27.614943    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:52:34.298443 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.298413    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:52:34.310395 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.310359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-tls-certs\") pod \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") "
Sep 17 21:52:34.310528 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.310424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnhvv\" (UniqueName: \"kubernetes.io/projected/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-kube-api-access-rnhvv\") pod \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\" (UID: \"b448b77a-50a2-47c3-898f-3cd66f6b6f9e\") "
Sep 17 21:52:34.312821 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.312774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b448b77a-50a2-47c3-898f-3cd66f6b6f9e" (UID: "b448b77a-50a2-47c3-898f-3cd66f6b6f9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:52:34.312821 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.312795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-kube-api-access-rnhvv" (OuterVolumeSpecName: "kube-api-access-rnhvv") pod "b448b77a-50a2-47c3-898f-3cd66f6b6f9e" (UID: "b448b77a-50a2-47c3-898f-3cd66f6b6f9e"). InnerVolumeSpecName "kube-api-access-rnhvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:52:34.411010 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.410981    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:52:34.411010 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.411008    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnhvv\" (UniqueName: \"kubernetes.io/projected/b448b77a-50a2-47c3-898f-3cd66f6b6f9e-kube-api-access-rnhvv\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:52:34.979165 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.979121    2577 generic.go:356] "Generic (PLEG): container finished" podID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerID="e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f" exitCode=0
Sep 17 21:52:34.979385 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.979180    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"
Sep 17 21:52:34.979385 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.979201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" event={"ID":"b448b77a-50a2-47c3-898f-3cd66f6b6f9e","Type":"ContainerDied","Data":"e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f"}
Sep 17 21:52:34.979385 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.979283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh" event={"ID":"b448b77a-50a2-47c3-898f-3cd66f6b6f9e","Type":"ContainerDied","Data":"6053aec4585e636c4c64b26c2ed24705e30e152e66eb05fe96c4c2693135397f"}
Sep 17 21:52:34.979385 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.979306    2577 scope.go:117] "RemoveContainer" containerID="e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f"
Sep 17 21:52:34.995654 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.989530    2577 scope.go:117] "RemoveContainer" containerID="e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f"
Sep 17 21:52:34.995654 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:52:34.994776    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f\": container with ID starting with e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f not found: ID does not exist" containerID="e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f"
Sep 17 21:52:34.995654 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:34.994817    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f"} err="failed to get container status \"e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f\": rpc error: code = NotFound desc = could not find container \"e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f\": container with ID starting with e9529fde2b8748365972025073d4837256a5483685ebbfa236016309f755e49f not found: ID does not exist"
Sep 17 21:52:35.011983 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:35.011958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"]
Sep 17 21:52:35.015696 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:35.015666    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5xpdh"]
Sep 17 21:52:35.717682 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:35.717646    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" path="/var/lib/kubelet/pods/b448b77a-50a2-47c3-898f-3cd66f6b6f9e/volumes"
Sep 17 21:52:39.464157 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:39.464120    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"]
Sep 17 21:52:43.950332 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.950289    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"]
Sep 17 21:52:43.950809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.950588    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main"
Sep 17 21:52:43.950809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.950598    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main"
Sep 17 21:52:43.950809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.950663    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b448b77a-50a2-47c3-898f-3cd66f6b6f9e" containerName="main"
Sep 17 21:52:43.953571 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.953551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:43.956074 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.956051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 21:52:43.957181 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.957162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 21:52:43.957305 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.957207    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 21:52:43.957421 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.957407    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-6djtx\""
Sep 17 21:52:43.957461 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.957436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-2xztj\""
Sep 17 21:52:43.969545 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.969522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"]
Sep 17 21:52:43.982501 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.982468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40f6df7-c048-4045-94f6-f38edf3c5c10-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:43.982618 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:43.982517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6cm\" (UniqueName: \"kubernetes.io/projected/d40f6df7-c048-4045-94f6-f38edf3c5c10-kube-api-access-rd6cm\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:44.083287 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:44.083231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40f6df7-c048-4045-94f6-f38edf3c5c10-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:44.083445 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:44.083390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6cm\" (UniqueName: \"kubernetes.io/projected/d40f6df7-c048-4045-94f6-f38edf3c5c10-kube-api-access-rd6cm\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:44.085916 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:44.085896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40f6df7-c048-4045-94f6-f38edf3c5c10-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:44.092879 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:44.092856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6cm\" (UniqueName: \"kubernetes.io/projected/d40f6df7-c048-4045-94f6-f38edf3c5c10-kube-api-access-rd6cm\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:44.264053 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:44.264028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:44.539168 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:44.539143    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"]
Sep 17 21:52:44.542154 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:52:44.542116    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40f6df7_c048_4045_94f6_f38edf3c5c10.slice/crio-98f5e02d8e5f98fd01228b66bfab1aa2911490da62c03169a684beb17a350374 WatchSource:0}: Error finding container 98f5e02d8e5f98fd01228b66bfab1aa2911490da62c03169a684beb17a350374: Status 404 returned error can't find the container with id 98f5e02d8e5f98fd01228b66bfab1aa2911490da62c03169a684beb17a350374
Sep 17 21:52:45.028454 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:45.028422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" event={"ID":"d40f6df7-c048-4045-94f6-f38edf3c5c10","Type":"ContainerStarted","Data":"dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9"}
Sep 17 21:52:45.028861 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:45.028480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" event={"ID":"d40f6df7-c048-4045-94f6-f38edf3c5c10","Type":"ContainerStarted","Data":"98f5e02d8e5f98fd01228b66bfab1aa2911490da62c03169a684beb17a350374"}
Sep 17 21:52:45.028861 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:45.028504    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:52:45.052335 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:45.052289    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" podStartSLOduration=2.052274211 podStartE2EDuration="2.052274211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:52:45.050439917 +0000 UTC m=+3803.869731902" watchObservedRunningTime="2026-09-17 21:52:45.052274211 +0000 UTC m=+3803.871566184"
Sep 17 21:52:46.420749 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:46.420705    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:52:54.489615 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:54.489573    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main" containerID="cri-o://9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b" gracePeriod=45
Sep 17 21:52:56.419838 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:52:56.419789    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:53:06.420233 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:06.420188    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:53:06.420858 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:06.420345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:53:16.035265 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:16.035212    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:53:16.420180 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:16.420098    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:53:17.735331 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:17.735297    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"]
Sep 17 21:53:24.729794 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:24.729766    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:53:24.789614 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:24.789581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fdedc-ac58-4831-b820-72fa3e20a902-tls-certs\") pod \"0d3fdedc-ac58-4831-b820-72fa3e20a902\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") "
Sep 17 21:53:24.789801 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:24.789642    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5vsk\" (UniqueName: \"kubernetes.io/projected/0d3fdedc-ac58-4831-b820-72fa3e20a902-kube-api-access-k5vsk\") pod \"0d3fdedc-ac58-4831-b820-72fa3e20a902\" (UID: \"0d3fdedc-ac58-4831-b820-72fa3e20a902\") "
Sep 17 21:53:24.791924 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:24.791900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3fdedc-ac58-4831-b820-72fa3e20a902-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d3fdedc-ac58-4831-b820-72fa3e20a902" (UID: "0d3fdedc-ac58-4831-b820-72fa3e20a902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:53:24.792033 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:24.791947    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3fdedc-ac58-4831-b820-72fa3e20a902-kube-api-access-k5vsk" (OuterVolumeSpecName: "kube-api-access-k5vsk") pod "0d3fdedc-ac58-4831-b820-72fa3e20a902" (UID: "0d3fdedc-ac58-4831-b820-72fa3e20a902"). InnerVolumeSpecName "kube-api-access-k5vsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:53:24.890623 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:24.890589    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fdedc-ac58-4831-b820-72fa3e20a902-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:53:24.890623 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:24.890617    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vsk\" (UniqueName: \"kubernetes.io/projected/0d3fdedc-ac58-4831-b820-72fa3e20a902-kube-api-access-k5vsk\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:53:25.220249 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.220148    2577 generic.go:356] "Generic (PLEG): container finished" podID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerID="9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b" exitCode=0
Sep 17 21:53:25.220249 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.220191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"
Sep 17 21:53:25.220249 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.220210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" event={"ID":"0d3fdedc-ac58-4831-b820-72fa3e20a902","Type":"ContainerDied","Data":"9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b"}
Sep 17 21:53:25.220502 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.220275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws" event={"ID":"0d3fdedc-ac58-4831-b820-72fa3e20a902","Type":"ContainerDied","Data":"d772f740b284f65ac3fd8394eed36db9dc46ad335a89923a6931272a6d9bfdcc"}
Sep 17 21:53:25.220502 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.220297    2577 scope.go:117] "RemoveContainer" containerID="9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b"
Sep 17 21:53:25.229554 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.229533    2577 scope.go:117] "RemoveContainer" containerID="9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b"
Sep 17 21:53:25.229833 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:53:25.229813    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b\": container with ID starting with 9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b not found: ID does not exist" containerID="9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b"
Sep 17 21:53:25.229880 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.229843    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b"} err="failed to get container status \"9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b\": rpc error: code = NotFound desc = could not find container \"9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b\": container with ID starting with 9b761c10ce14d5053f0fd67f5a62557a5f70676d01c8591e05b316410b8ecb0b not found: ID does not exist"
Sep 17 21:53:25.242069 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.242036    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"]
Sep 17 21:53:25.247993 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.247967    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-7b5ws"]
Sep 17 21:53:25.717366 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:25.717332    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" path="/var/lib/kubelet/pods/0d3fdedc-ac58-4831-b820-72fa3e20a902/volumes"
Sep 17 21:53:26.034577 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:26.034537    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:53:32.755633 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:32.755578    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main" containerID="cri-o://dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9" gracePeriod=45
Sep 17 21:53:35.360843 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.360805    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"]
Sep 17 21:53:35.361346 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.361300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main"
Sep 17 21:53:35.361346 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.361318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main"
Sep 17 21:53:35.361511 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.361410    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3fdedc-ac58-4831-b820-72fa3e20a902" containerName="main"
Sep 17 21:53:35.366201 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.366177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.369969 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.369946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 21:53:35.370232 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.369961    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 21:53:35.370363 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.369961    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-vghvz\""
Sep 17 21:53:35.370420 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.370104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-w6qcl\""
Sep 17 21:53:35.370475 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.370117    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 21:53:35.376532 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.376510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"]
Sep 17 21:53:35.476070 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.476038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c66689b-2a1c-496c-ba05-959ff7819556-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.476263 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.476114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kslx\" (UniqueName: \"kubernetes.io/projected/4c66689b-2a1c-496c-ba05-959ff7819556-kube-api-access-2kslx\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.576758 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.576714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kslx\" (UniqueName: \"kubernetes.io/projected/4c66689b-2a1c-496c-ba05-959ff7819556-kube-api-access-2kslx\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.576758 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.576759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c66689b-2a1c-496c-ba05-959ff7819556-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.579379 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.579358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c66689b-2a1c-496c-ba05-959ff7819556-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.586130 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.586102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kslx\" (UniqueName: \"kubernetes.io/projected/4c66689b-2a1c-496c-ba05-959ff7819556-kube-api-access-2kslx\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.678219 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.678129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:35.951813 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:35.951792    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"]
Sep 17 21:53:35.954106 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:53:35.954070    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c66689b_2a1c_496c_ba05_959ff7819556.slice/crio-1d2f25999478eff5f0032566f5a1dde46beb98792835de7741fa0c938c3f57ef WatchSource:0}: Error finding container 1d2f25999478eff5f0032566f5a1dde46beb98792835de7741fa0c938c3f57ef: Status 404 returned error can't find the container with id 1d2f25999478eff5f0032566f5a1dde46beb98792835de7741fa0c938c3f57ef
Sep 17 21:53:36.035280 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:36.035222    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:53:36.278547 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:36.278512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" event={"ID":"4c66689b-2a1c-496c-ba05-959ff7819556","Type":"ContainerStarted","Data":"8a88346c42dee463817768c108262f99a77c532942c98a1aa3d245d273c67883"}
Sep 17 21:53:36.278731 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:36.278563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" event={"ID":"4c66689b-2a1c-496c-ba05-959ff7819556","Type":"ContainerStarted","Data":"1d2f25999478eff5f0032566f5a1dde46beb98792835de7741fa0c938c3f57ef"}
Sep 17 21:53:36.278731 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:36.278630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:53:36.299533 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:36.299464    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" podStartSLOduration=1.2994444600000001 podStartE2EDuration="1.29944446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:53:36.298440364 +0000 UTC m=+3855.117732326" watchObservedRunningTime="2026-09-17 21:53:36.29944446 +0000 UTC m=+3855.118736439"
Sep 17 21:53:46.035049 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:46.035012    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:53:46.035461 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:46.035112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:53:56.034499 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:53:56.034462    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:54:03.001436 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.001404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:54:03.101999 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.101902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd6cm\" (UniqueName: \"kubernetes.io/projected/d40f6df7-c048-4045-94f6-f38edf3c5c10-kube-api-access-rd6cm\") pod \"d40f6df7-c048-4045-94f6-f38edf3c5c10\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") "
Sep 17 21:54:03.102139 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.102006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40f6df7-c048-4045-94f6-f38edf3c5c10-tls-certs\") pod \"d40f6df7-c048-4045-94f6-f38edf3c5c10\" (UID: \"d40f6df7-c048-4045-94f6-f38edf3c5c10\") "
Sep 17 21:54:03.104329 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.104299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40f6df7-c048-4045-94f6-f38edf3c5c10-kube-api-access-rd6cm" (OuterVolumeSpecName: "kube-api-access-rd6cm") pod "d40f6df7-c048-4045-94f6-f38edf3c5c10" (UID: "d40f6df7-c048-4045-94f6-f38edf3c5c10"). InnerVolumeSpecName "kube-api-access-rd6cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:54:03.104329 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.104316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40f6df7-c048-4045-94f6-f38edf3c5c10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d40f6df7-c048-4045-94f6-f38edf3c5c10" (UID: "d40f6df7-c048-4045-94f6-f38edf3c5c10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:54:03.202574 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.202538    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd6cm\" (UniqueName: \"kubernetes.io/projected/d40f6df7-c048-4045-94f6-f38edf3c5c10-kube-api-access-rd6cm\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:54:03.202574 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.202567    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40f6df7-c048-4045-94f6-f38edf3c5c10-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:54:03.407809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.407713    2577 generic.go:356] "Generic (PLEG): container finished" podID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerID="dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9" exitCode=0
Sep 17 21:54:03.407809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.407759    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"
Sep 17 21:54:03.407809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.407800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" event={"ID":"d40f6df7-c048-4045-94f6-f38edf3c5c10","Type":"ContainerDied","Data":"dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9"}
Sep 17 21:54:03.408026 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.407874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b" event={"ID":"d40f6df7-c048-4045-94f6-f38edf3c5c10","Type":"ContainerDied","Data":"98f5e02d8e5f98fd01228b66bfab1aa2911490da62c03169a684beb17a350374"}
Sep 17 21:54:03.408026 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.407896    2577 scope.go:117] "RemoveContainer" containerID="dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9"
Sep 17 21:54:03.419018 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.418969    2577 scope.go:117] "RemoveContainer" containerID="dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9"
Sep 17 21:54:03.419588 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:54:03.419565    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9\": container with ID starting with dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9 not found: ID does not exist" containerID="dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9"
Sep 17 21:54:03.419678 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.419595    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9"} err="failed to get container status \"dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9\": rpc error: code = NotFound desc = could not find container \"dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9\": container with ID starting with dd47165a7ab093010457e4f52a0bfdd4166104a1635076032dd747eb3de18fd9 not found: ID does not exist"
Sep 17 21:54:03.430183 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.430148    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"]
Sep 17 21:54:03.431961 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.431939    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9b566b"]
Sep 17 21:54:03.717780 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:03.717701    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" path="/var/lib/kubelet/pods/d40f6df7-c048-4045-94f6-f38edf3c5c10/volumes"
Sep 17 21:54:07.286400 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:07.286370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:54:21.826684 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.826642    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"]
Sep 17 21:54:21.827072 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.826943    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main"
Sep 17 21:54:21.827072 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.826954    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main"
Sep 17 21:54:21.827072 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.827016    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40f6df7-c048-4045-94f6-f38edf3c5c10" containerName="main"
Sep 17 21:54:21.837580 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.837551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:21.841190 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.841160    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"]
Sep 17 21:54:21.841381 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.841361    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-22r9q\""
Sep 17 21:54:21.841502 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.841190    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 21:54:21.841697 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.841325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-bzmdz\""
Sep 17 21:54:21.841802 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.841307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 21:54:21.841873 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.841681    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 21:54:21.937755 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.937715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79280a-1b59-41e9-bbea-6e86f22b83d6-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:21.937966 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:21.937810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plndb\" (UniqueName: \"kubernetes.io/projected/6b79280a-1b59-41e9-bbea-6e86f22b83d6-kube-api-access-plndb\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:22.038852 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.038801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plndb\" (UniqueName: \"kubernetes.io/projected/6b79280a-1b59-41e9-bbea-6e86f22b83d6-kube-api-access-plndb\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:22.039035 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.038882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79280a-1b59-41e9-bbea-6e86f22b83d6-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:22.041716 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.041685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79280a-1b59-41e9-bbea-6e86f22b83d6-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:22.047324 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.047296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plndb\" (UniqueName: \"kubernetes.io/projected/6b79280a-1b59-41e9-bbea-6e86f22b83d6-kube-api-access-plndb\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:22.149476 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.149375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:22.260865 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.260834    2577 scope.go:117] "RemoveContainer" containerID="425e5a54ac0b43fdeb56357f77815f1d43f40452124342fc25929708ad9510e3"
Sep 17 21:54:22.425211 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.425133    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"]
Sep 17 21:54:22.429064 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:54:22.429035    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b79280a_1b59_41e9_bbea_6e86f22b83d6.slice/crio-9f00ba80775211cbb5610d4f003e72aeb8fa93f86ee3ee84d069b7197b6570a1 WatchSource:0}: Error finding container 9f00ba80775211cbb5610d4f003e72aeb8fa93f86ee3ee84d069b7197b6570a1: Status 404 returned error can't find the container with id 9f00ba80775211cbb5610d4f003e72aeb8fa93f86ee3ee84d069b7197b6570a1
Sep 17 21:54:22.503745 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.503709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" event={"ID":"6b79280a-1b59-41e9-bbea-6e86f22b83d6","Type":"ContainerStarted","Data":"0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8"}
Sep 17 21:54:22.503931 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.503762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" event={"ID":"6b79280a-1b59-41e9-bbea-6e86f22b83d6","Type":"ContainerStarted","Data":"9f00ba80775211cbb5610d4f003e72aeb8fa93f86ee3ee84d069b7197b6570a1"}
Sep 17 21:54:22.503931 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.503837    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:54:22.527334 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:22.527270    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" podStartSLOduration=1.527220493 podStartE2EDuration="1.527220493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:54:22.525884589 +0000 UTC m=+3901.345176564" watchObservedRunningTime="2026-09-17 21:54:22.527220493 +0000 UTC m=+3901.346512456"
Sep 17 21:54:53.510613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:54:53.510583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:55:04.931442 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:04.931403    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"]
Sep 17 21:55:13.510589 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:13.510510    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:55:19.951861 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:19.951800    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main" containerID="cri-o://0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8" gracePeriod=45
Sep 17 21:55:23.510262 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:23.510195    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:55:33.510038 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:33.509999    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:55:33.510515 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:33.510102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:55:43.509862 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:43.509826    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:55:50.208831 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.208806    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:55:50.365840 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.365808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plndb\" (UniqueName: \"kubernetes.io/projected/6b79280a-1b59-41e9-bbea-6e86f22b83d6-kube-api-access-plndb\") pod \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") "
Sep 17 21:55:50.366027 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.365884    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79280a-1b59-41e9-bbea-6e86f22b83d6-tls-certs\") pod \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\" (UID: \"6b79280a-1b59-41e9-bbea-6e86f22b83d6\") "
Sep 17 21:55:50.368278 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.368225    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b79280a-1b59-41e9-bbea-6e86f22b83d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b79280a-1b59-41e9-bbea-6e86f22b83d6" (UID: "6b79280a-1b59-41e9-bbea-6e86f22b83d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:55:50.368392 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.368277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b79280a-1b59-41e9-bbea-6e86f22b83d6-kube-api-access-plndb" (OuterVolumeSpecName: "kube-api-access-plndb") pod "6b79280a-1b59-41e9-bbea-6e86f22b83d6" (UID: "6b79280a-1b59-41e9-bbea-6e86f22b83d6"). InnerVolumeSpecName "kube-api-access-plndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:55:50.466670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.466633    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79280a-1b59-41e9-bbea-6e86f22b83d6-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:55:50.466670 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.466664    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plndb\" (UniqueName: \"kubernetes.io/projected/6b79280a-1b59-41e9-bbea-6e86f22b83d6-kube-api-access-plndb\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:55:50.933388 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.933351    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerID="0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8" exitCode=0
Sep 17 21:55:50.933570 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.933402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"
Sep 17 21:55:50.933570 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.933412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" event={"ID":"6b79280a-1b59-41e9-bbea-6e86f22b83d6","Type":"ContainerDied","Data":"0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8"}
Sep 17 21:55:50.933570 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.933480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w" event={"ID":"6b79280a-1b59-41e9-bbea-6e86f22b83d6","Type":"ContainerDied","Data":"9f00ba80775211cbb5610d4f003e72aeb8fa93f86ee3ee84d069b7197b6570a1"}
Sep 17 21:55:50.933570 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.933503    2577 scope.go:117] "RemoveContainer" containerID="0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8"
Sep 17 21:55:50.942992 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.942975    2577 scope.go:117] "RemoveContainer" containerID="0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8"
Sep 17 21:55:50.943311 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:55:50.943256    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8\": container with ID starting with 0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8 not found: ID does not exist" containerID="0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8"
Sep 17 21:55:50.943311 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.943284    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8"} err="failed to get container status \"0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8\": rpc error: code = NotFound desc = could not find container \"0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8\": container with ID starting with 0260ecaed31a5a91b2ada101fae90b5f8a09c585a40dc3dc56d40f06f075dee8 not found: ID does not exist"
Sep 17 21:55:50.957273 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.957230    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"]
Sep 17 21:55:50.960849 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:50.960827    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-stq7w"]
Sep 17 21:55:51.718336 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:51.718304    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" path="/var/lib/kubelet/pods/6b79280a-1b59-41e9-bbea-6e86f22b83d6/volumes"
Sep 17 21:55:58.036177 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:55:58.036139    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"]
Sep 17 21:56:07.285675 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:07.285635    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:56:13.057280 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:13.057224    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main" containerID="cri-o://8a88346c42dee463817768c108262f99a77c532942c98a1aa3d245d273c67883" gracePeriod=45
Sep 17 21:56:17.286252 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:17.286208    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:56:27.286217 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:27.286177    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:56:27.286613 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:27.286289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:56:37.286401 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:37.286353    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:56:43.175778 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.175744    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c66689b-2a1c-496c-ba05-959ff7819556" containerID="8a88346c42dee463817768c108262f99a77c532942c98a1aa3d245d273c67883" exitCode=0
Sep 17 21:56:43.176096 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.175817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" event={"ID":"4c66689b-2a1c-496c-ba05-959ff7819556","Type":"ContainerDied","Data":"8a88346c42dee463817768c108262f99a77c532942c98a1aa3d245d273c67883"}
Sep 17 21:56:43.300701 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.300675    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:56:43.378812 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.378725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c66689b-2a1c-496c-ba05-959ff7819556-tls-certs\") pod \"4c66689b-2a1c-496c-ba05-959ff7819556\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") "
Sep 17 21:56:43.378973 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.378815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kslx\" (UniqueName: \"kubernetes.io/projected/4c66689b-2a1c-496c-ba05-959ff7819556-kube-api-access-2kslx\") pod \"4c66689b-2a1c-496c-ba05-959ff7819556\" (UID: \"4c66689b-2a1c-496c-ba05-959ff7819556\") "
Sep 17 21:56:43.381124 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.381098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c66689b-2a1c-496c-ba05-959ff7819556-kube-api-access-2kslx" (OuterVolumeSpecName: "kube-api-access-2kslx") pod "4c66689b-2a1c-496c-ba05-959ff7819556" (UID: "4c66689b-2a1c-496c-ba05-959ff7819556"). InnerVolumeSpecName "kube-api-access-2kslx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:56:43.381221 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.381131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c66689b-2a1c-496c-ba05-959ff7819556-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c66689b-2a1c-496c-ba05-959ff7819556" (UID: "4c66689b-2a1c-496c-ba05-959ff7819556"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:56:43.480258 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.480200    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c66689b-2a1c-496c-ba05-959ff7819556-tls-certs\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:56:43.480258 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:43.480263    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kslx\" (UniqueName: \"kubernetes.io/projected/4c66689b-2a1c-496c-ba05-959ff7819556-kube-api-access-2kslx\") on node \"ip-10-0-134-35.ec2.internal\" DevicePath \"\""
Sep 17 21:56:44.181401 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:44.181369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn" event={"ID":"4c66689b-2a1c-496c-ba05-959ff7819556","Type":"ContainerDied","Data":"1d2f25999478eff5f0032566f5a1dde46beb98792835de7741fa0c938c3f57ef"}
Sep 17 21:56:44.181836 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:44.181432    2577 scope.go:117] "RemoveContainer" containerID="8a88346c42dee463817768c108262f99a77c532942c98a1aa3d245d273c67883"
Sep 17 21:56:44.181836 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:44.181384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"
Sep 17 21:56:44.202534 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:44.202302    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"]
Sep 17 21:56:44.208113 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:44.208089    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ddssn"]
Sep 17 21:56:45.716934 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:56:45.716900    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" path="/var/lib/kubelet/pods/4c66689b-2a1c-496c-ba05-959ff7819556/volumes"
Sep 17 21:57:28.068819 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:28.068784    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vhdj6_5c4c5349-cd3f-4bbe-829e-a9110d7f82e5/discovery/0.log"
Sep 17 21:57:28.988663 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:28.988625    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vhdj6_5c4c5349-cd3f-4bbe-829e-a9110d7f82e5/discovery/0.log"
Sep 17 21:57:30.111510 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:30.111476    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-wfp95_d308ceef-55f2-49f6-ac50-01f31e1108cc/manager/0.log"
Sep 17 21:57:30.124022 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:30.123999    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-vv29f_0d401026-f73a-41c2-ac49-694e36219faf/manager/0.log"
Sep 17 21:57:30.290323 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:30.290295    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-294bq_ff910491-41c6-4b8c-afa2-f212e1b8452e/manager/0.log"
Sep 17 21:57:37.449867 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:37.449835    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lsbm5_843bd172-4a4b-4285-8810-a1edee197936/global-pull-secret-syncer/0.log"
Sep 17 21:57:37.558710 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:37.558679    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nzjvb_72932197-8c1b-4d17-9e63-7cc9b1de8316/konnectivity-agent/0.log"
Sep 17 21:57:37.648015 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:37.647977    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-35.ec2.internal_43a2109c14934db412a36a19c6e1114f/haproxy/0.log"
Sep 17 21:57:42.513626 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:42.513581    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-wfp95_d308ceef-55f2-49f6-ac50-01f31e1108cc/manager/0.log"
Sep 17 21:57:42.540946 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:42.540917    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-vv29f_0d401026-f73a-41c2-ac49-694e36219faf/manager/0.log"
Sep 17 21:57:42.843056 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:42.842970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-294bq_ff910491-41c6-4b8c-afa2-f212e1b8452e/manager/0.log"
Sep 17 21:57:44.434376 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:44.434346    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9jzjk_620dcb8e-480f-4815-8152-695cdf9e9ef0/node-exporter/0.log"
Sep 17 21:57:44.454584 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:44.454550    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9jzjk_620dcb8e-480f-4815-8152-695cdf9e9ef0/kube-rbac-proxy/0.log"
Sep 17 21:57:44.480474 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:44.480445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9jzjk_620dcb8e-480f-4815-8152-695cdf9e9ef0/init-textfile/0.log"
Sep 17 21:57:47.264576 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.264533    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"]
Sep 17 21:57:47.265092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.264956    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main"
Sep 17 21:57:47.265092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.264975    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main"
Sep 17 21:57:47.265092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.265005    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main"
Sep 17 21:57:47.265092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.265013    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main"
Sep 17 21:57:47.265092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.265079    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b79280a-1b59-41e9-bbea-6e86f22b83d6" containerName="main"
Sep 17 21:57:47.265092 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.265089    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c66689b-2a1c-496c-ba05-959ff7819556" containerName="main"
Sep 17 21:57:47.268102 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.268070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.271160 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:57:47.271126    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-134-35.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-j8tzf\": no relationship found between node 'ip-10-0-134-35.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-j8tzf\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 21:57:47.271160 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:57:47.271131    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"perf-node-gather-daemonset-5xhl7\" is forbidden: User \"system:node:ip-10-0-134-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j8tzf\": no relationship found between node 'ip-10-0-134-35.ec2.internal' and this object" podUID="ca51ef0f-0f02-4721-a878-8d2a921346e7" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.271351 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:57:47.271191    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-4wxnw\" is forbidden: User \"system:node:ip-10-0-134-35.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-must-gather-j8tzf\": no relationship found between node 'ip-10-0-134-35.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-j8tzf\"/\"default-dockercfg-4wxnw\"" type="*v1.Secret"
Sep 17 21:57:47.271351 ip-10-0-134-35 kubenswrapper[2577]: E0917 21:57:47.271193    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-134-35.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-j8tzf\": no relationship found between node 'ip-10-0-134-35.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-j8tzf\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 21:57:47.290153 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.290120    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"]
Sep 17 21:57:47.371156 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.371116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdxv\" (UniqueName: \"kubernetes.io/projected/ca51ef0f-0f02-4721-a878-8d2a921346e7-kube-api-access-vvdxv\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.371384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.371165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-lib-modules\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.371384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.371185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-podres\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.371384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.371206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-proc\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.371384 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.371300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-sys\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.472551 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdxv\" (UniqueName: \"kubernetes.io/projected/ca51ef0f-0f02-4721-a878-8d2a921346e7-kube-api-access-vvdxv\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.472769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-lib-modules\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.472769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-podres\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.472769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-proc\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.472769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-sys\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.472769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-sys\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.472769 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-lib-modules\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.473041 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-proc\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.473041 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.472754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca51ef0f-0f02-4721-a878-8d2a921346e7-podres\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:47.564862 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:47.564780    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cff44857c-q59vq_1f467411-4bd4-457f-823f-467c9b019925/console/0.log"
Sep 17 21:57:48.305538 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:48.305503    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8tzf\"/\"kube-root-ca.crt\""
Sep 17 21:57:48.793824 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:48.793793    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8tzf\"/\"default-dockercfg-4wxnw\""
Sep 17 21:57:48.806473 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:48.806444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8tzf\"/\"openshift-service-ca.crt\""
Sep 17 21:57:48.812737 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:48.812716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdxv\" (UniqueName: \"kubernetes.io/projected/ca51ef0f-0f02-4721-a878-8d2a921346e7-kube-api-access-vvdxv\") pod \"perf-node-gather-daemonset-5xhl7\" (UID: \"ca51ef0f-0f02-4721-a878-8d2a921346e7\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:49.077459 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.077369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:49.246406 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.246377    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bb52p_6c18ebf4-2af7-4849-b66f-243015f1b7a4/dns/0.log"
Sep 17 21:57:49.267454 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.267424    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bb52p_6c18ebf4-2af7-4849-b66f-243015f1b7a4/kube-rbac-proxy/0.log"
Sep 17 21:57:49.351306 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.351273    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"]
Sep 17 21:57:49.353910 ip-10-0-134-35 kubenswrapper[2577]: W0917 21:57:49.353875    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podca51ef0f_0f02_4721_a878_8d2a921346e7.slice/crio-ddf8def9eb9bbacf98d027c8f75c4622410ac18aa13b29963bbba034e254a2ed WatchSource:0}: Error finding container ddf8def9eb9bbacf98d027c8f75c4622410ac18aa13b29963bbba034e254a2ed: Status 404 returned error can't find the container with id ddf8def9eb9bbacf98d027c8f75c4622410ac18aa13b29963bbba034e254a2ed
Sep 17 21:57:49.375563 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.375537    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h5ltl_141c4051-4a21-4899-8871-eb7c00e693a2/dns-node-resolver/0.log"
Sep 17 21:57:49.476855 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.476820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7" event={"ID":"ca51ef0f-0f02-4721-a878-8d2a921346e7","Type":"ContainerStarted","Data":"d4fbbe9c194f787646a53185ced22e3051c37de0a4ab1e007de5e9675b5e7960"}
Sep 17 21:57:49.477032 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.476888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:49.477032 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.476911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7" event={"ID":"ca51ef0f-0f02-4721-a878-8d2a921346e7","Type":"ContainerStarted","Data":"ddf8def9eb9bbacf98d027c8f75c4622410ac18aa13b29963bbba034e254a2ed"}
Sep 17 21:57:49.493530 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.493483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7" podStartSLOduration=2.49346709 podStartE2EDuration="2.49346709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:57:49.491670558 +0000 UTC m=+4108.310962520" watchObservedRunningTime="2026-09-17 21:57:49.49346709 +0000 UTC m=+4108.312759052"
Sep 17 21:57:49.918990 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.918955    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-554db46f86-nqcg6_4b241ce8-0d29-4e6f-8713-fff56fcf9774/registry/0.log"
Sep 17 21:57:49.992261 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:49.992221    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m6lqg_49392876-888a-4002-a542-faa186562b69/node-ca/0.log"
Sep 17 21:57:51.048503 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:51.048460    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vhdj6_5c4c5349-cd3f-4bbe-829e-a9110d7f82e5/discovery/0.log"
Sep 17 21:57:51.764357 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:51.764315    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mmt5q_a6630ca4-9c28-4f87-b38b-363fa2a0c764/serve-healthcheck-canary/0.log"
Sep 17 21:57:52.436777 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:52.436743    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hknz7_b6f16adf-204a-4830-b44b-aac227bb8712/kube-rbac-proxy/0.log"
Sep 17 21:57:52.459809 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:52.459780    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hknz7_b6f16adf-204a-4830-b44b-aac227bb8712/exporter/0.log"
Sep 17 21:57:52.483149 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:52.483120    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hknz7_b6f16adf-204a-4830-b44b-aac227bb8712/extractor/0.log"
Sep 17 21:57:55.492094 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:55.492069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-5xhl7"
Sep 17 21:57:56.374032 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:56.373984    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-66f4b8957f-zzbqz_e90ca681-208a-4549-ba1e-0bfa197927dd/manager/0.log"
Sep 17 21:57:56.822939 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:56.822895    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-79ffffc8b5-8mg29_c6277daf-bc60-4af8-8bf5-f100e77cfab8/manager/0.log"
Sep 17 21:57:56.841335 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:56.841310    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-x6zsn_9cc6197c-695e-445d-926a-bf47af035bcc/server/0.log"
Sep 17 21:57:57.233447 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:57.233404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vhdj6_5c4c5349-cd3f-4bbe-829e-a9110d7f82e5/discovery/0.log"
Sep 17 21:57:57.920368 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:57.920321    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-4qdtq_03d5b668-1f6c-4b01-aa8a-1c4b889abd97/manager/0.log"
Sep 17 21:57:58.013640 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:58.013604    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-tdrlh_8478205f-ddfe-4517-9c02-1aaf8ae2ecf9/seaweedfs/0.log"
Sep 17 21:57:58.584112 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:57:58.584081    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vhdj6_5c4c5349-cd3f-4bbe-829e-a9110d7f82e5/discovery/0.log"
Sep 17 21:58:04.766393 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:04.766359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfgtm_580f0aac-e36f-422d-b6dd-959e3b57d61f/kube-multus-additional-cni-plugins/0.log"
Sep 17 21:58:04.788180 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:04.788153    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfgtm_580f0aac-e36f-422d-b6dd-959e3b57d61f/egress-router-binary-copy/0.log"
Sep 17 21:58:04.807315 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:04.807271    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfgtm_580f0aac-e36f-422d-b6dd-959e3b57d61f/cni-plugins/0.log"
Sep 17 21:58:04.826784 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:04.826752    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfgtm_580f0aac-e36f-422d-b6dd-959e3b57d61f/bond-cni-plugin/0.log"
Sep 17 21:58:04.845416 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:04.845389    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfgtm_580f0aac-e36f-422d-b6dd-959e3b57d61f/routeoverride-cni/0.log"
Sep 17 21:58:04.863823 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:04.863798    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfgtm_580f0aac-e36f-422d-b6dd-959e3b57d61f/whereabouts-cni-bincopy/0.log"
Sep 17 21:58:04.882655 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:04.882629    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hfgtm_580f0aac-e36f-422d-b6dd-959e3b57d61f/whereabouts-cni/0.log"
Sep 17 21:58:05.077724 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:05.077646    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fv4xg_8ed1c2f5-88fd-453f-9b58-82b7e81fc60b/kube-multus/0.log"
Sep 17 21:58:05.188531 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:05.188500    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tl6x_2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b/network-metrics-daemon/0.log"
Sep 17 21:58:05.203530 ip-10-0-134-35 kubenswrapper[2577]: I0917 21:58:05.203476    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tl6x_2e6dbb90-da4d-4e9f-b3f0-c24624e0e08b/kube-rbac-proxy/0.log"