Sep 17 17:31:42.126605 ip-10-0-135-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:31:42.126619 ip-10-0-135-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:31:42.126629 ip-10-0-135-84 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:31:42.126827 ip-10-0-135-84 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:31:52.173684 ip-10-0-135-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:31:52.173701 ip-10-0-135-84 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot abbc3e80010b4ec2aabb652a69ad2ac7 --
Sep 17 17:33:58.490299 ip-10-0-135-84 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:34:02.267974 ip-10-0-135-84 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:02.267974 ip-10-0-135-84 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:34:02.267974 ip-10-0-135-84 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:02.267974 ip-10-0-135-84 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:34:02.267974 ip-10-0-135-84 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:02.268942 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.268072    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:34:02.273245 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273228    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:02.273245 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273245    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273249    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273253    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273257    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273260    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273264    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273266    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273269    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273272    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273275    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273278    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273281    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273283    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273286    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273289    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273292    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273295    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273298    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273306    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273308    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:02.273318 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273311    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273314    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273317    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273320    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273323    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273326    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273329    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273332    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273335    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273338    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273340    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273343    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273346    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273349    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273352    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273354    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273357    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273367    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273369    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:02.273832 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273372    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273376    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273379    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273381    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273384    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273387    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273389    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273392    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273394    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273397    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273400    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273402    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273405    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273408    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273411    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273417    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273421    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273425    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273428    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:02.274363 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273431    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273434    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273437    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273440    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273443    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273446    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273449    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273452    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273455    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273460    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273465    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273468    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273472    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273475    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273477    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273480    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273483    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273486    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273488    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273491    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:02.274867 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273494    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273496    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273501    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273504    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273507    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273510    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273513    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273517    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273520    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273524    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273527    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273529    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273532    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273535    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273538    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273541    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273543    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273546    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273549    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.273552    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:02.275443 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274301    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274309    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274313    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274317    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274320    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274323    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274326    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274329    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274332    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274335    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274338    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274340    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274343    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274346    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274349    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274351    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274354    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274357    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274360    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274364    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:02.275974 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274367    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274370    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274373    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274376    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274379    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274382    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274385    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274388    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274390    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274393    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274396    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274399    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274402    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274404    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274407    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274410    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274413    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274415    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274419    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274422    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:02.276534 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274424    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274427    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274430    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274433    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274435    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274438    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274441    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274443    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274447    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274450    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274452    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274456    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274459    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274461    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274464    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274467    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274470    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274473    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274475    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:02.277074 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274478    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274482    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274486    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274489    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274492    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274494    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274497    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274499    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274503    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274506    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274509    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274511    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274515    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274517    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274520    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274523    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274526    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274529    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274532    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274535    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:02.277612 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274538    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274541    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274545    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274550    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274554    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274559    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274562    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274564    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274567    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274570    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274573    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274576    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274579    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274582    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274584    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274587    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274590    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274593    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274595    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:02.278155 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.274598    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275020    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275026    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275030    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275033    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275037    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275039    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275043    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275045    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275048    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275051    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275054    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275057    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275060    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275063    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275066    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275069    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275073    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275077    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275079    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:02.278660 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275082    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275085    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275088    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275091    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275093    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275096    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275099    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275102    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275104    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275107    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275110    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275113    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275116    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275118    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275122    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275125    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275127    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275130    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275134    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275137    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:02.279257 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275139    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275142    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275147    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275150    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275153    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275158    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275162    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275165    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275168    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275171    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275174    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275177    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275181    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275184    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275188    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275191    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275194    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275197    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275199    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:02.279816 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275202    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275205    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275207    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275210    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275214    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275217    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275220    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275226    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275228    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275231    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275234    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275238    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275241    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275244    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275246    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275249    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275252    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275255    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275257    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275260    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:02.280329 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275263    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275266    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275269    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275271    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275274    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275277    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275280    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275282    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275285    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275288    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275290    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275293    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275296    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275299    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275302    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275304    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275307    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275310    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275313    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:02.280893 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275317    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.275319    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275400    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275436    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275442    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275447    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275452    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275456    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275460    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275465    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275468    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275472    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275476    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275480    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275484    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275487    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275490    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275493    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275496    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275499    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275504    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275507    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275510    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275513    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:34:02.281409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275517    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275521    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275524    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275527    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275531    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275534    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275537    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275540    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275545    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275548    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275552    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275555    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275558    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275562    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275565    2566 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275569    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275574    2566 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275577    2566 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275580    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275583    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275587    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275591    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275594    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275597    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275600    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:34:02.282036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275604    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275607    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275610    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275613    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275616    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275619    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275622    2566 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275626    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275630    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275633    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275636    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275640    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275643    2566 flags.go:64] FLAG: --help="false"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275646    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275649    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275654    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275657    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275661    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275665    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275668    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275671    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275674    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275677    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:34:02.282748 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275681    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275685    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275688    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275691    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275694    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275698    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275701    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275704    2566 flags.go:64] FLAG: --lock-file=""
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275707    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275710    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275713    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275719    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275722    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275725    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275729    2566 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275732    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275736    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275739    2566 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275742    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275749    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275753    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275757    2566 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275760    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275764    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275768    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:34:02.283466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275772    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275775    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275778    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275781    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275789    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275792    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275796    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275799    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275803    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275809    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275812    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275815    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275819    2566 flags.go:64] FLAG: --port="10250"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275822    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275825    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05164f7be91149de6"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275828    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275831    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275835    2566 flags.go:64] FLAG: --register-node="true"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275838    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275842    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275845    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275849    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275852    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275856    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275859    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:34:02.284147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275862    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275865    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275868    2566 flags.go:64] FLAG: --runonce="false"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275871    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275875    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275878    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275882    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275885    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275888    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275891    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275895    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275898    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275901    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275904    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275907    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275910    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275914    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275917    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275923    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275927    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275930    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275935    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275938    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275941    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275944    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:34:02.284844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275947    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275951    2566 flags.go:64] FLAG: --v="2"
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275955    2566 flags.go:64] FLAG: --version="false"
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275959    2566 flags.go:64] FLAG: --vmodule=""
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275963    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.275966    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276100    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276104    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276108    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276111    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276113    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276116    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276119    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276123    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276126    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276129    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276132    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276135    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276138    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276140    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276143    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:02.285579 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276146    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276149    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276152    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276155    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276158    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276163    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276165    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276168    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276171    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276174    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276177    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276179    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276182    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276185    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276188    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276191    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276194    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276197    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276199    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276202    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:02.286507 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276205    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276208    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276211    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276213    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276216    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276219    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276222    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276227    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276230    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276233    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276236    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276239    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276242    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276246    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276250    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276253    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276257    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276261    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276264    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:02.287223 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276267    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276269    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276272    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276275    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276278    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276281    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276284    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276286    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276289    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276292    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276295    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276298    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276301    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276303    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276306    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276309    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276312    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276315    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276318    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276321    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:02.287740 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276324    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276327    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276329    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276332    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276335    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276339    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276343    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276346    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276349    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276352    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276357    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276360    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276363    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276365    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276368    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276371    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276374    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276377    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276380    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:02.288285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276382    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276385    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276388    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276391    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276393    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.276396    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.276919    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.284407    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.284424    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.284444    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.284451    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284512    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284517    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284521    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284524    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:02.288789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284527    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284530    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284533    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284536    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284539    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284542    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284545    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284548    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284550    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284553    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284557    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284560    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284563    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284566    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284569    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284572    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284575    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284578    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284581    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284584    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:02.289224 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284586    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284590    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284593    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284597    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284602    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284606    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284609    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284612    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284616    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284619    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284622    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284625    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284629    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284632    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284635    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284638    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284641    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284644    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284647    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:02.289766 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284650    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284653    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284656    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284659    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284662    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284664    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284667    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284670    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284673    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284676    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284679    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284682    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284685    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284688    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284692    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284695    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284698    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284702    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284705    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284709    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:02.290275 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284713    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284715    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284718    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284721    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284725    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284730    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284734    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284737    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284740    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284742    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284745    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284748    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284751    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284753    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284756    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284759    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284762    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284765    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284768    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:02.290820 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284770    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284773    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284777    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284780    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284783    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284786    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284788    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284792    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284795    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284797    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284800    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284803    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284806    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284809    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284812    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284815    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:02.291388 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284818    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.284823    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284945    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284951    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284954    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284957    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284961    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284964    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284969    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284973    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284976    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284979    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284982    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.284999    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:02.291814 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285002    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285005    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285008    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285011    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285014    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285017    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285020    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285023    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285026    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285034    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285037    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285040    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285043    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285046    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285050    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285052    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285056    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285059    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285062    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285065    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:02.292274 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285067    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285070    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285073    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285076    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285079    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285082    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285084    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285087    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285090    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285093    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285095    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285098    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285101    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285104    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285106    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285109    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285112    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285115    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285118    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285121    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:02.292804 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285124    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285126    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285129    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285132    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285135    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285138    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285140    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285143    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285146    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285149    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285152    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285155    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285158    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285161    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285163    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285166    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285169    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285171    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285175    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285179    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:02.293357 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285182    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285185    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285187    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285190    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285193    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285196    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285198    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285201    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285204    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285207    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285210    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285213    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285216    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285219    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285221    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285224    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285227    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285230    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285233    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285236    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:02.293884 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285238    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285241    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285244    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285247    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285250    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285253    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:02.285255    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.285261    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.285384    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.289511    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.290518    2566 server.go:1019] "Starting client certificate rotation"
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.290632    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.290674    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:02.294503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.293082    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:02.294948 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.294281    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:02.299503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.299486    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:34:02.301609 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.301591    2566 log.go:25] "Validated CRI v1 image API"
Sep 17 17:34:02.302487 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.302471    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:34:02.305081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.305060    2566 fs.go:135] Filesystem UUIDs: map[42103db2-dd39-4640-8a70-70b8a8e2b6c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 df275ba4-3ede-429e-897c-30757d69fab1:/dev/nvme0n1p3]
Sep 17 17:34:02.305163 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.305079    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:34:02.309576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.309460    2566 manager.go:216] Machine: {Timestamp:2026-09-17 17:34:02.308954203 +0000 UTC m=+2.168545553 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107210 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c2f57e8765ee4de14baea0075f646 SystemUUID:ec2c2f57-e876-5ee4-de14-baea0075f646 BootID:abbc3e80-010b-4ec2-aabb-652a69ad2ac7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:92:ea:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:92:ea:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:24:2d:b8:c4:7b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:34:02.309576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.309566    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:34:02.309751 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.309711    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:34:02.311625 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.311593    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:34:02.311791 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.311627    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:34:02.312965 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.312951    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:34:02.313052 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.312974    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:34:02.313052 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.312997    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:02.313052 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.313014    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:02.313194 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.313103    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:34:02.316823 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.316806    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:02.316966 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.316955    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:34:02.317066 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.317012    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:34:02.317066 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.317025    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:34:02.317066 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.317045    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:34:02.317066 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.317058    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:34:02.317201 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.317072    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:34:02.317485 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.317470    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:02.317537 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.317498    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:02.329419 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.329399    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:34:02.332217 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.332199    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:02.334000 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.333934    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:34:02.334000 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.333967    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:34:02.335217 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335201    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335225    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335235    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335243    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335256    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335265    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335286    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335294    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:34:02.335309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335304    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:34:02.335600 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335314    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:34:02.335600 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335329    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:34:02.335600 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335338    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:34:02.335600 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335376    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:34:02.335600 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.335386    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:34:02.338715 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.338684    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:34:02.338824 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.338753    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:34:02.340408 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.340390    2566 server.go:1268] "Started kubelet"
Sep 17 17:34:02.340497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.340457    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:34:02.340549 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.340485    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:34:02.340595 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.340571    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:34:02.341253 ip-10-0-135-84 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:34:02.341719 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.341490    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:34:02.342096 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.341961    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:34:02.345183 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.345158    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:34:02.348119 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348104    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:34:02.348188 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348110    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:02.348627 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.348285    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:02.348742 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348723    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:34:02.348843 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348769    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:34:02.348843 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348745    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:34:02.348952 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348881    2566 factory.go:55] Registering systemd factory
Sep 17 17:34:02.348952 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348898    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:34:02.348952 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348921    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:34:02.348952 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348947    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:34:02.349157 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.348756    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:34:02.349157 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.349091    2566 factory.go:153] Registering CRI-O factory
Sep 17 17:34:02.349157 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.349114    2566 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:34:02.349271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.349205    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:34:02.349271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.349226    2566 factory.go:103] Registering Raw factory
Sep 17 17:34:02.349271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.349244    2566 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:34:02.349700 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.349682    2566 manager.go:349] Starting recovery of all containers
Sep 17 17:34:02.360208 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.360184    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:34:02.360292 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.360234    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:34:02.374602 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.374571    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8v2s5"
Sep 17 17:34:02.375628 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.375607    2566 manager.go:354] Recovery completed
Sep 17 17:34:02.380912 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.380892    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8v2s5"
Sep 17 17:34:02.381081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.381066    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:02.382715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.382700    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:02.382770 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.382727    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:02.382770 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.382737    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:02.383410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.383396    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:02.383410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.383403    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:34:02.383513 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.383414    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:34:02.383513 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.383432    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:02.384135 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.384119    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:02.384194 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.384147    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:02.384194 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.384164    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:02.384623 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.384606    2566 policy_none.go:49] "None policy: Start"
Sep 17 17:34:02.384623 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.384626    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:34:02.384736 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.384637    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:34:02.385705 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.385692    2566 policy_none.go:47] "Start"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.423537    2566 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.423581    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.423594    2566 server.go:87] "Starting device plugin registration server"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.423825    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.423847    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.423973    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.424043    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.424049    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.424542    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:34:02.431722 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.424594    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:02.499168 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.499145    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:34:02.500406 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.500387    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:34:02.500481 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.500445    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:34:02.500481 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.500476    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:34:02.500558 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.500523    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:34:02.502421 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.502404    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:02.524726 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.524663    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:02.525504 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.525470    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:02.525504 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.525505    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:02.525695 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.525524    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:02.525695 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.525555    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.538490 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.538468    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.538583 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.538492    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-84.ec2.internal\": node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:02.550432 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.550415    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:02.601329 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.601305    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal"]
Sep 17 17:34:02.601401 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.601362    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:02.602735 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.602717    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:02.602813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.602750    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:02.602813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.602765    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:02.604078 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604064    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:02.604186 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.604234 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604214    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:02.604868 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604851    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:02.604962 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604856    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:02.604962 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604882    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:02.604962 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604897    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:02.604962 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604898    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:02.604962 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.604965    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:02.606311 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.606297    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.606374 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.606321    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:02.607537 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.607521    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:02.607611 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.607549    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:02.607611 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.607563    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:02.637445 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.637425    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-84.ec2.internal\" not found" node="ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.641594 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.641581    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-84.ec2.internal\" not found" node="ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.650353 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.650335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46bebf7b7c741450a2f3f54893c5a937-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal\" (UID: \"46bebf7b7c741450a2f3f54893c5a937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.650411 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.650362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46bebf7b7c741450a2f3f54893c5a937-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal\" (UID: \"46bebf7b7c741450a2f3f54893c5a937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.650411 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.650379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af7109a8d4ffd33a7b7c4fc7c44515af-config\") pod \"kube-apiserver-proxy-ip-10-0-135-84.ec2.internal\" (UID: \"af7109a8d4ffd33a7b7c4fc7c44515af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.651364 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.651351    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:02.751436 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.751414    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:02.751509 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.751481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af7109a8d4ffd33a7b7c4fc7c44515af-config\") pod \"kube-apiserver-proxy-ip-10-0-135-84.ec2.internal\" (UID: \"af7109a8d4ffd33a7b7c4fc7c44515af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.751509 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.751503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46bebf7b7c741450a2f3f54893c5a937-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal\" (UID: \"46bebf7b7c741450a2f3f54893c5a937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.751576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.751521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46bebf7b7c741450a2f3f54893c5a937-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal\" (UID: \"46bebf7b7c741450a2f3f54893c5a937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.751576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.751551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46bebf7b7c741450a2f3f54893c5a937-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal\" (UID: \"46bebf7b7c741450a2f3f54893c5a937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.751576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.751564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af7109a8d4ffd33a7b7c4fc7c44515af-config\") pod \"kube-apiserver-proxy-ip-10-0-135-84.ec2.internal\" (UID: \"af7109a8d4ffd33a7b7c4fc7c44515af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.751661 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.751580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46bebf7b7c741450a2f3f54893c5a937-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal\" (UID: \"46bebf7b7c741450a2f3f54893c5a937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.851764 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.851705    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:02.940834 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.940811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.946062 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:02.946040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:02.952372 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:02.952354    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:03.052774 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:03.052749    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:03.153176 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:03.153114    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:03.253609 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:03.253579    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:03.290738 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.290713    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:34:03.291169 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.290879    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:03.342129 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.341996    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:03.354542 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:03.354522    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:03.385667 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.385618    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:29:02 +0000 UTC" deadline="2028-02-27 03:10:26.517013929 +0000 UTC"
Sep 17 17:34:03.385749 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.385666    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12657h36m23.131351532s"
Sep 17 17:34:03.455082 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:03.455057    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:03.477601 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:03.477563    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7109a8d4ffd33a7b7c4fc7c44515af.slice/crio-4911d92ac6a6ae0427ebdf743372801419e99165b9bc4e9320809bfc2bee709f WatchSource:0}: Error finding container 4911d92ac6a6ae0427ebdf743372801419e99165b9bc4e9320809bfc2bee709f: Status 404 returned error can't find the container with id 4911d92ac6a6ae0427ebdf743372801419e99165b9bc4e9320809bfc2bee709f
Sep 17 17:34:03.477877 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:03.477858    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bebf7b7c741450a2f3f54893c5a937.slice/crio-20e099174e25e5f88e89e5cdf1d03304d8ac5813f0dfeaa9a6284fb37dcd78dc WatchSource:0}: Error finding container 20e099174e25e5f88e89e5cdf1d03304d8ac5813f0dfeaa9a6284fb37dcd78dc: Status 404 returned error can't find the container with id 20e099174e25e5f88e89e5cdf1d03304d8ac5813f0dfeaa9a6284fb37dcd78dc
Sep 17 17:34:03.482831 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.482810    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:03.503568 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.503526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal" event={"ID":"46bebf7b7c741450a2f3f54893c5a937","Type":"ContainerStarted","Data":"20e099174e25e5f88e89e5cdf1d03304d8ac5813f0dfeaa9a6284fb37dcd78dc"}
Sep 17 17:34:03.506669 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.506645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal" event={"ID":"af7109a8d4ffd33a7b7c4fc7c44515af","Type":"ContainerStarted","Data":"4911d92ac6a6ae0427ebdf743372801419e99165b9bc4e9320809bfc2bee709f"}
Sep 17 17:34:03.555968 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:03.555937    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-84.ec2.internal\" not found"
Sep 17 17:34:03.643104 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.643084    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:03.648859 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.648839    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:03.661448 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.661431    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:03.662478 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.662466    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal"
Sep 17 17:34:03.673851 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.673835    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:03.879894 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:03.879698    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:04.109927 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.109720    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:04.318895 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.318809    2566 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:04.327053 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.327027    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:04.327523 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.327489    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal","openshift-multus/multus-additional-cni-plugins-dqjpg","openshift-multus/network-metrics-daemon-4q9vq","openshift-network-diagnostics/network-check-target-gbxbc","kube-system/konnectivity-agent-tpz2m","kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp","openshift-cluster-node-tuning-operator/tuned-5mg5t","openshift-multus/multus-qx4hz","openshift-network-operator/iptables-alerter-lw2p5","openshift-ovn-kubernetes/ovnkube-node-gpmbx","kube-system/global-pull-secret-syncer-8gpqj","openshift-dns/node-resolver-npxnb","openshift-image-registry/node-ca-h4cs9"]
Sep 17 17:34:04.329342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.329323    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.330791 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.330767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.331735 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.331718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:04.331834 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.331783    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:04.332251 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.332082    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:04.333281 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.332666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:04.333281 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.332734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-psdff\""
Sep 17 17:34:04.333281 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.332884    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:04.333281 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.333269    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47bfd\""
Sep 17 17:34:04.333536 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.333504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:04.333590 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.333561    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:04.333922 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.333676    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:04.333922 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.333752    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:04.334496 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.334479    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.336226 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.336210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.336822 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.336660    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:04.336822 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.336714    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crl2l\""
Sep 17 17:34:04.336822 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.336819    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:04.337394 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.337367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.338887 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.338735    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:04.339062 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.339034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:04.339147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.339063    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:04.339482 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.339464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jzbss\""
Sep 17 17:34:04.339596 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.339578    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:04.339929 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.339797    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:04.339929 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.339817    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:04.340125 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.340106    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:04.340125 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.340117    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v5x5h\""
Sep 17 17:34:04.344601 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.344381    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.345809 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.345785    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.346566 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.346546    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5p685\""
Sep 17 17:34:04.346692 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.346670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:04.346812 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.346796    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:04.347434 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.347063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.347951 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.347936    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:04.348474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.348451    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:04.348711 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.348676    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:04.348954 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.348934    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wghfp\""
Sep 17 17:34:04.349039 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.348898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.349632 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.349612    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:04.350651 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.350624    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:04.350891 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.350875    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:04.351501 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.351475    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjx7d\""
Sep 17 17:34:04.352745 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.352724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:04.353010 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.352990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.353100 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.353023    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:04.353100 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.353043    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:04.353378 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.353063    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:04.354813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.354259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:04.354813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.354297    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sh2gh\""
Sep 17 17:34:04.354813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.354310    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:04.354813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.354523    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:04.359127 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-slash\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.359224 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-var-lib-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.359224 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-tmp-dir\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.359224 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:04.359224 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-cni-binary-copy\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.359224 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkrh\" (UniqueName: \"kubernetes.io/projected/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-kube-api-access-klkrh\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-lib-modules\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-hostroot\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-kubelet\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm88m\" (UniqueName: \"kubernetes.io/projected/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-kube-api-access-dm88m\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-registration-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:04.359466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xh9n\" (UniqueName: \"kubernetes.io/projected/3ed98aec-15bb-4997-89c2-6606d5c377be-kube-api-access-5xh9n\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-log-socket\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-serviceca\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysctl-d\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-systemd\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-tuned\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-socket-dir-parent\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc060665-a4d6-4f7f-90be-faaf39e1a20f-iptables-alerter-script\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-run-netns\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd9nc\" (UniqueName: \"kubernetes.io/projected/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-kube-api-access-zd9nc\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.359838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-multus-certs\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-etc-kubernetes\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-ovn\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovnkube-config\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.359983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-socket-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-cni-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-cni-multus\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-kubelet\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-run\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-etc-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-cni-bin\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4eccb060-886d-4453-aa0f-2160986e0827-dbus\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-device-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-cnibin\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-daemon-config\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.360507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39058041-be1f-48b3-b2b2-5310ac76a7ad-tmp\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-systemd-units\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4eccb060-886d-4453-aa0f-2160986e0827-kubelet-config\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tkng\" (UniqueName: \"kubernetes.io/projected/31adc1af-ab71-49d0-9a26-dae9735bb36d-kube-api-access-4tkng\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-system-cni-dir\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysconfig\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-systemd\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-node-log\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovnkube-script-lib\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/541d06dd-ac25-4278-9164-bd5eb107c596-konnectivity-ca\") pod \"konnectivity-agent-tpz2m\" (UID: \"541d06dd-ac25-4278-9164-bd5eb107c596\") " pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-env-overrides\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjb79\" (UniqueName: \"kubernetes.io/projected/717c1fea-058a-46b5-89c1-7b17e3b77db4-kube-api-access-rjb79\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-cni-netd\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-hosts-file\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.361239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/541d06dd-ac25-4278-9164-bd5eb107c596-agent-certs\") pod \"konnectivity-agent-tpz2m\" (UID: \"541d06dd-ac25-4278-9164-bd5eb107c596\") " pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-system-cni-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-k8s-cni-cncf-io\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-cni-bin\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-conf-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-sys-fs\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgffk\" (UniqueName: \"kubernetes.io/projected/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-kube-api-access-tgffk\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-cnibin\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-kubernetes\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysctl-conf\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-var-lib-kubelet\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.360987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-host\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-modprobe-d\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-sys\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4xb\" (UniqueName: \"kubernetes.io/projected/fc060665-a4d6-4f7f-90be-faaf39e1a20f-kube-api-access-9r4xb\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovn-node-metrics-cert\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.361827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-host\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.362543 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-netns\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.362543 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgpn\" (UniqueName: \"kubernetes.io/projected/39058041-be1f-48b3-b2b2-5310ac76a7ad-kube-api-access-kpgpn\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.362543 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.361228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc060665-a4d6-4f7f-90be-faaf39e1a20f-host-slash\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.363765 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.363669    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:04.407556 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.407537    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mj8vb"
Sep 17 17:34:04.418241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.418220    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mj8vb"
Sep 17 17:34:04.450221 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.450203    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:04.461631 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.461719 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-socket-dir-parent\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.461719 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.461832 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc060665-a4d6-4f7f-90be-faaf39e1a20f-iptables-alerter-script\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.461832 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.461832 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-socket-dir-parent\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.461832 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.461832 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-run-netns\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9nc\" (UniqueName: \"kubernetes.io/projected/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-kube-api-access-zd9nc\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-multus-certs\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-etc-kubernetes\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-ovn\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovnkube-config\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-socket-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.461994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-cni-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-cni-multus\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-kubelet\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-run\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-etc-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-cni-bin\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4eccb060-886d-4453-aa0f-2160986e0827-dbus\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-device-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-cnibin\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-daemon-config\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39058041-be1f-48b3-b2b2-5310ac76a7ad-tmp\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-systemd-units\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4eccb060-886d-4453-aa0f-2160986e0827-kubelet-config\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tkng\" (UniqueName: \"kubernetes.io/projected/31adc1af-ab71-49d0-9a26-dae9735bb36d-kube-api-access-4tkng\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-system-cni-dir\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc060665-a4d6-4f7f-90be-faaf39e1a20f-iptables-alerter-script\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysconfig\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-systemd\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-run-netns\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-node-log\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.462820 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-node-log\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-etc-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-systemd-units\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4eccb060-886d-4453-aa0f-2160986e0827-kubelet-config\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-multus-certs\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-etc-kubernetes\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462804    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-system-cni-dir\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysconfig\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-systemd\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-cni-multus\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovnkube-config\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-kubelet\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovnkube-script-lib\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-run\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/541d06dd-ac25-4278-9164-bd5eb107c596-konnectivity-ca\") pod \"konnectivity-agent-tpz2m\" (UID: \"541d06dd-ac25-4278-9164-bd5eb107c596\") " pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.463622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-env-overrides\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjb79\" (UniqueName: \"kubernetes.io/projected/717c1fea-058a-46b5-89c1-7b17e3b77db4-kube-api-access-rjb79\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-socket-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-cni-netd\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4eccb060-886d-4453-aa0f-2160986e0827-dbus\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-hosts-file\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-cni-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/541d06dd-ac25-4278-9164-bd5eb107c596-agent-certs\") pod \"konnectivity-agent-tpz2m\" (UID: \"541d06dd-ac25-4278-9164-bd5eb107c596\") " pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-system-cni-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-k8s-cni-cncf-io\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-cni-bin\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-conf-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-sys-fs\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgffk\" (UniqueName: \"kubernetes.io/projected/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-kube-api-access-tgffk\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-cnibin\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.464392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-kubernetes\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysctl-conf\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-var-lib-kubelet\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-host\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-modprobe-d\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-sys\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4xb\" (UniqueName: \"kubernetes.io/projected/fc060665-a4d6-4f7f-90be-faaf39e1a20f-kube-api-access-9r4xb\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovn-node-metrics-cert\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-host\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-netns\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgpn\" (UniqueName: \"kubernetes.io/projected/39058041-be1f-48b3-b2b2-5310ac76a7ad-kube-api-access-kpgpn\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc060665-a4d6-4f7f-90be-faaf39e1a20f-host-slash\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-slash\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-var-lib-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-tmp-dir\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovnkube-script-lib\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.465185 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-cni-netd\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-cni-binary-copy\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klkrh\" (UniqueName: \"kubernetes.io/projected/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-kube-api-access-klkrh\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464283    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717c1fea-058a-46b5-89c1-7b17e3b77db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-lib-modules\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-hostroot\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-kubelet\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm88m\" (UniqueName: \"kubernetes.io/projected/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-kube-api-access-dm88m\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-daemon-config\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.462810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-ovn\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-registration-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xh9n\" (UniqueName: \"kubernetes.io/projected/3ed98aec-15bb-4997-89c2-6606d5c377be-kube-api-access-5xh9n\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-log-socket\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466172 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-serviceca\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysctl-d\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-systemd\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-tuned\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/541d06dd-ac25-4278-9164-bd5eb107c596-konnectivity-ca\") pod \"konnectivity-agent-tpz2m\" (UID: \"541d06dd-ac25-4278-9164-bd5eb107c596\") " pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-cni-binary-copy\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-var-lib-kubelet\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-env-overrides\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-hosts-file\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.464972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-host\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-run-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-lib-modules\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-modprobe-d\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-hostroot\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-sys\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465192    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-kubelet\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-slash\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.466968 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-device-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-host\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465560    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-log-socket\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-tmp-dir\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-registration-dir\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-multus-conf-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-var-lib-openvswitch\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-system-cni-dir\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.465695    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.463641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-host-cni-bin\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-var-lib-cni-bin\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-k8s-cni-cncf-io\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.465781    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs podName:3ed98aec-15bb-4997-89c2-6606d5c377be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:04.96574584 +0000 UTC m=+4.825337200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs") pod "network-metrics-daemon-4q9vq" (UID: "3ed98aec-15bb-4997-89c2-6606d5c377be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-cnibin\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717c1fea-058a-46b5-89c1-7b17e3b77db4-cnibin\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-sys-fs\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-kubernetes\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.465975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-serviceca\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.467708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.466003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-systemd\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.466037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31adc1af-ab71-49d0-9a26-dae9735bb36d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.466043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysctl-conf\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.466068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-host-run-netns\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.466072    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.466123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-sysctl-d\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.466166    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret podName:4eccb060-886d-4453-aa0f-2160986e0827 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:04.966149754 +0000 UTC m=+4.825741104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret") pod "global-pull-secret-syncer-8gpqj" (UID: "4eccb060-886d-4453-aa0f-2160986e0827") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.466184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc060665-a4d6-4f7f-90be-faaf39e1a20f-host-slash\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.467833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39058041-be1f-48b3-b2b2-5310ac76a7ad-tmp\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.467857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39058041-be1f-48b3-b2b2-5310ac76a7ad-etc-tuned\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.468551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.468325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-ovn-node-metrics-cert\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.468983 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.468734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/541d06dd-ac25-4278-9164-bd5eb107c596-agent-certs\") pod \"konnectivity-agent-tpz2m\" (UID: \"541d06dd-ac25-4278-9164-bd5eb107c596\") " pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.482244 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.479789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkrh\" (UniqueName: \"kubernetes.io/projected/ba2ba32b-2d80-4def-8d77-2899b37d9f0b-kube-api-access-klkrh\") pod \"multus-qx4hz\" (UID: \"ba2ba32b-2d80-4def-8d77-2899b37d9f0b\") " pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.482244 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.480278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tkng\" (UniqueName: \"kubernetes.io/projected/31adc1af-ab71-49d0-9a26-dae9735bb36d-kube-api-access-4tkng\") pod \"aws-ebs-csi-driver-node-rnnkp\" (UID: \"31adc1af-ab71-49d0-9a26-dae9735bb36d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.482244 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.480823    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:04.482244 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.480847    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:04.482244 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.480860    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ngqlp for pod openshift-network-diagnostics/network-check-target-gbxbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:04.482244 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.480940    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp podName:8d8f3843-88d6-45fb-9b0a-4ae823a6a064 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:04.980921904 +0000 UTC m=+4.840513254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ngqlp" (UniqueName: "kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp") pod "network-check-target-gbxbc" (UID: "8d8f3843-88d6-45fb-9b0a-4ae823a6a064") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:04.482969 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.482942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9nc\" (UniqueName: \"kubernetes.io/projected/72e8ef0e-4a19-4a80-b7db-057ffafcfd50-kube-api-access-zd9nc\") pod \"node-resolver-npxnb\" (UID: \"72e8ef0e-4a19-4a80-b7db-057ffafcfd50\") " pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.483168 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.483014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjb79\" (UniqueName: \"kubernetes.io/projected/717c1fea-058a-46b5-89c1-7b17e3b77db4-kube-api-access-rjb79\") pod \"multus-additional-cni-plugins-dqjpg\" (UID: \"717c1fea-058a-46b5-89c1-7b17e3b77db4\") " pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.490366 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.490347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgffk\" (UniqueName: \"kubernetes.io/projected/14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d-kube-api-access-tgffk\") pod \"node-ca-h4cs9\" (UID: \"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d\") " pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.493220 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.493198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgpn\" (UniqueName: \"kubernetes.io/projected/39058041-be1f-48b3-b2b2-5310ac76a7ad-kube-api-access-kpgpn\") pod \"tuned-5mg5t\" (UID: \"39058041-be1f-48b3-b2b2-5310ac76a7ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.493314 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.493299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4xb\" (UniqueName: \"kubernetes.io/projected/fc060665-a4d6-4f7f-90be-faaf39e1a20f-kube-api-access-9r4xb\") pod \"iptables-alerter-lw2p5\" (UID: \"fc060665-a4d6-4f7f-90be-faaf39e1a20f\") " pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.493798 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.493783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xh9n\" (UniqueName: \"kubernetes.io/projected/3ed98aec-15bb-4997-89c2-6606d5c377be-kube-api-access-5xh9n\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:04.493856 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.493837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm88m\" (UniqueName: \"kubernetes.io/projected/be1e098d-32d9-456d-b2e8-cf856f7f7dd9-kube-api-access-dm88m\") pod \"ovnkube-node-gpmbx\" (UID: \"be1e098d-32d9-456d-b2e8-cf856f7f7dd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.653797 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.653733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp"
Sep 17 17:34:04.658098 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.658074    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npxnb"
Sep 17 17:34:04.669084 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.669065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:04.673615 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.673597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h4cs9"
Sep 17 17:34:04.679258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.679240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qx4hz"
Sep 17 17:34:04.685747 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.685724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5mg5t"
Sep 17 17:34:04.691211 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.691193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dqjpg"
Sep 17 17:34:04.698937 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.698905    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lw2p5"
Sep 17 17:34:04.703445 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.703426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:04.969200 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.969126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:04.969200 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:04.969188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:04.969409 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.969289    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:04.969409 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.969293    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:04.969409 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.969351    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs podName:3ed98aec-15bb-4997-89c2-6606d5c377be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:05.96933325 +0000 UTC m=+5.828924584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs") pod "network-metrics-daemon-4q9vq" (UID: "3ed98aec-15bb-4997-89c2-6606d5c377be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:04.969409 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:04.969368    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret podName:4eccb060-886d-4453-aa0f-2160986e0827 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:05.969361645 +0000 UTC m=+5.828952978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret") pod "global-pull-secret-syncer-8gpqj" (UID: "4eccb060-886d-4453-aa0f-2160986e0827") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:05.069779 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.069738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:05.069959 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.069907    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:05.069959 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.069940    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:05.069959 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.069949    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ngqlp for pod openshift-network-diagnostics/network-check-target-gbxbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:05.070071 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.069999    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp podName:8d8f3843-88d6-45fb-9b0a-4ae823a6a064 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:06.069985944 +0000 UTC m=+5.929577279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngqlp" (UniqueName: "kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp") pod "network-check-target-gbxbc" (UID: "8d8f3843-88d6-45fb-9b0a-4ae823a6a064") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:05.215122 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.215099    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717c1fea_058a_46b5_89c1_7b17e3b77db4.slice/crio-7272bf89ea2c279e9e1e42fff3cf2ff9cf04657b2333ab8f91c8d697199cf22c WatchSource:0}: Error finding container 7272bf89ea2c279e9e1e42fff3cf2ff9cf04657b2333ab8f91c8d697199cf22c: Status 404 returned error can't find the container with id 7272bf89ea2c279e9e1e42fff3cf2ff9cf04657b2333ab8f91c8d697199cf22c
Sep 17 17:34:05.216196 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.216176    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31adc1af_ab71_49d0_9a26_dae9735bb36d.slice/crio-cffc470dae5ee62ab38457755e2c72fbe77db39b94c0cd035c97f0ad572f94eb WatchSource:0}: Error finding container cffc470dae5ee62ab38457755e2c72fbe77db39b94c0cd035c97f0ad572f94eb: Status 404 returned error can't find the container with id cffc470dae5ee62ab38457755e2c72fbe77db39b94c0cd035c97f0ad572f94eb
Sep 17 17:34:05.217254 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.217225    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1e098d_32d9_456d_b2e8_cf856f7f7dd9.slice/crio-24e290f5f2241a77023e000df099113ec37d957d180d1bb3828d2b55b7a2ba41 WatchSource:0}: Error finding container 24e290f5f2241a77023e000df099113ec37d957d180d1bb3828d2b55b7a2ba41: Status 404 returned error can't find the container with id 24e290f5f2241a77023e000df099113ec37d957d180d1bb3828d2b55b7a2ba41
Sep 17 17:34:05.221374 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.221352    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e8ef0e_4a19_4a80_b7db_057ffafcfd50.slice/crio-42c7a008a815914244dc48d8be3ff235bb7b11739bf9dc38dddbea8e68f49990 WatchSource:0}: Error finding container 42c7a008a815914244dc48d8be3ff235bb7b11739bf9dc38dddbea8e68f49990: Status 404 returned error can't find the container with id 42c7a008a815914244dc48d8be3ff235bb7b11739bf9dc38dddbea8e68f49990
Sep 17 17:34:05.222187 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.222164    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f32c3f_67dd_4ab0_9e2a_1ea09fae7f3d.slice/crio-aea57edcfb537dc3fcafefa6c936121f6d2971c19f8781f83397b6250f7629f4 WatchSource:0}: Error finding container aea57edcfb537dc3fcafefa6c936121f6d2971c19f8781f83397b6250f7629f4: Status 404 returned error can't find the container with id aea57edcfb537dc3fcafefa6c936121f6d2971c19f8781f83397b6250f7629f4
Sep 17 17:34:05.223789 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.223691    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2ba32b_2d80_4def_8d77_2899b37d9f0b.slice/crio-c20427f624e1a88fc302756a0ecd27f5c922ce8619abaca0034e1ee5e06d9e4e WatchSource:0}: Error finding container c20427f624e1a88fc302756a0ecd27f5c922ce8619abaca0034e1ee5e06d9e4e: Status 404 returned error can't find the container with id c20427f624e1a88fc302756a0ecd27f5c922ce8619abaca0034e1ee5e06d9e4e
Sep 17 17:34:05.224335 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.224307    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541d06dd_ac25_4278_9164_bd5eb107c596.slice/crio-941a048582463e9ecf4f45bf2d7205ce0cec27a0b284a526f177af985d1ccf33 WatchSource:0}: Error finding container 941a048582463e9ecf4f45bf2d7205ce0cec27a0b284a526f177af985d1ccf33: Status 404 returned error can't find the container with id 941a048582463e9ecf4f45bf2d7205ce0cec27a0b284a526f177af985d1ccf33
Sep 17 17:34:05.225502 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.225475    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc060665_a4d6_4f7f_90be_faaf39e1a20f.slice/crio-fa11ed3639318830aee3a2fad572ddc835270f4279407297493995e72afa5377 WatchSource:0}: Error finding container fa11ed3639318830aee3a2fad572ddc835270f4279407297493995e72afa5377: Status 404 returned error can't find the container with id fa11ed3639318830aee3a2fad572ddc835270f4279407297493995e72afa5377
Sep 17 17:34:05.226072 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:05.226052    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39058041_be1f_48b3_b2b2_5310ac76a7ad.slice/crio-35be1866598209470b86c7b2391e586426296f05e42f0d55f99701cbb6f976cc WatchSource:0}: Error finding container 35be1866598209470b86c7b2391e586426296f05e42f0d55f99701cbb6f976cc: Status 404 returned error can't find the container with id 35be1866598209470b86c7b2391e586426296f05e42f0d55f99701cbb6f976cc
Sep 17 17:34:05.420454 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.420404    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:04 +0000 UTC" deadline="2028-03-08 18:49:45.842592274 +0000 UTC"
Sep 17 17:34:05.420454 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.420448    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12913h15m40.42214608s"
Sep 17 17:34:05.510592 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.510532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5mg5t" event={"ID":"39058041-be1f-48b3-b2b2-5310ac76a7ad","Type":"ContainerStarted","Data":"35be1866598209470b86c7b2391e586426296f05e42f0d55f99701cbb6f976cc"}
Sep 17 17:34:05.511672 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.511653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lw2p5" event={"ID":"fc060665-a4d6-4f7f-90be-faaf39e1a20f","Type":"ContainerStarted","Data":"fa11ed3639318830aee3a2fad572ddc835270f4279407297493995e72afa5377"}
Sep 17 17:34:05.512704 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.512685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h4cs9" event={"ID":"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d","Type":"ContainerStarted","Data":"aea57edcfb537dc3fcafefa6c936121f6d2971c19f8781f83397b6250f7629f4"}
Sep 17 17:34:05.514572 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.514549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal" event={"ID":"af7109a8d4ffd33a7b7c4fc7c44515af","Type":"ContainerStarted","Data":"e29957966686c24962dc0df29a70a3192150e1e90607832350df5ed2b2c66279"}
Sep 17 17:34:05.515681 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.515660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpz2m" event={"ID":"541d06dd-ac25-4278-9164-bd5eb107c596","Type":"ContainerStarted","Data":"941a048582463e9ecf4f45bf2d7205ce0cec27a0b284a526f177af985d1ccf33"}
Sep 17 17:34:05.516749 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.516718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qx4hz" event={"ID":"ba2ba32b-2d80-4def-8d77-2899b37d9f0b","Type":"ContainerStarted","Data":"c20427f624e1a88fc302756a0ecd27f5c922ce8619abaca0034e1ee5e06d9e4e"}
Sep 17 17:34:05.517745 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.517726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npxnb" event={"ID":"72e8ef0e-4a19-4a80-b7db-057ffafcfd50","Type":"ContainerStarted","Data":"42c7a008a815914244dc48d8be3ff235bb7b11739bf9dc38dddbea8e68f49990"}
Sep 17 17:34:05.518744 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.518720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"24e290f5f2241a77023e000df099113ec37d957d180d1bb3828d2b55b7a2ba41"}
Sep 17 17:34:05.519881 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.519860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp" event={"ID":"31adc1af-ab71-49d0-9a26-dae9735bb36d","Type":"ContainerStarted","Data":"cffc470dae5ee62ab38457755e2c72fbe77db39b94c0cd035c97f0ad572f94eb"}
Sep 17 17:34:05.521154 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.521131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerStarted","Data":"7272bf89ea2c279e9e1e42fff3cf2ff9cf04657b2333ab8f91c8d697199cf22c"}
Sep 17 17:34:05.522746 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.522726    2566 generic.go:356] "Generic (PLEG): container finished" podID="46bebf7b7c741450a2f3f54893c5a937" containerID="1744f9e0482ca6505a8f039cd90b662db2bb44f465c8f4a655226fe79a99838f" exitCode=0
Sep 17 17:34:05.522827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.522752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal" event={"ID":"46bebf7b7c741450a2f3f54893c5a937","Type":"ContainerDied","Data":"1744f9e0482ca6505a8f039cd90b662db2bb44f465c8f4a655226fe79a99838f"}
Sep 17 17:34:05.552593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.552266    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-84.ec2.internal" podStartSLOduration=2.55224957 podStartE2EDuration="2.55224957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:05.533726342 +0000 UTC m=+5.393317701" watchObservedRunningTime="2026-09-17 17:34:05.55224957 +0000 UTC m=+5.411840959"
Sep 17 17:34:05.976693 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.976208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:05.976693 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.976488    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:05.976693 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:05.976506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:05.976693 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.976565    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret podName:4eccb060-886d-4453-aa0f-2160986e0827 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:07.976543247 +0000 UTC m=+7.836134587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret") pod "global-pull-secret-syncer-8gpqj" (UID: "4eccb060-886d-4453-aa0f-2160986e0827") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:05.976693 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.976628    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:05.976693 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:05.976675    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs podName:3ed98aec-15bb-4997-89c2-6606d5c377be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:07.976660128 +0000 UTC m=+7.836251462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs") pod "network-metrics-daemon-4q9vq" (UID: "3ed98aec-15bb-4997-89c2-6606d5c377be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:06.077603 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:06.077564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:06.077767 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:06.077752    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:06.077842 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:06.077772    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:06.077842 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:06.077785    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ngqlp for pod openshift-network-diagnostics/network-check-target-gbxbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:06.077979 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:06.077846    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp podName:8d8f3843-88d6-45fb-9b0a-4ae823a6a064 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.077826437 +0000 UTC m=+7.937417785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngqlp" (UniqueName: "kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp") pod "network-check-target-gbxbc" (UID: "8d8f3843-88d6-45fb-9b0a-4ae823a6a064") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:06.421274 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:06.421157    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:04 +0000 UTC" deadline="2028-06-10 14:26:39.719872158 +0000 UTC"
Sep 17 17:34:06.421274 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:06.421196    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15164h52m33.298680424s"
Sep 17 17:34:06.504020 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:06.503988    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:06.504176 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:06.504129    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:06.504541 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:06.504521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:06.504640 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:06.504618    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:06.504697 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:06.504676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:06.504788 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:06.504740    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:06.531900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:06.531872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal" event={"ID":"46bebf7b7c741450a2f3f54893c5a937","Type":"ContainerStarted","Data":"a9ca71fc03df0206fed7329d8b4f3d94a317b77a18135cc00ec0e28f461f36cc"}
Sep 17 17:34:08.000507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:08.000465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:08.001009 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:08.000550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:08.001009 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.000684    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.001009 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.000747    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs podName:3ed98aec-15bb-4997-89c2-6606d5c377be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:12.000727655 +0000 UTC m=+11.860319014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs") pod "network-metrics-daemon-4q9vq" (UID: "3ed98aec-15bb-4997-89c2-6606d5c377be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.001235 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.001214    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.001288 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.001274    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret podName:4eccb060-886d-4453-aa0f-2160986e0827 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:12.001258677 +0000 UTC m=+11.860850028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret") pod "global-pull-secret-syncer-8gpqj" (UID: "4eccb060-886d-4453-aa0f-2160986e0827") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.101784 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:08.101744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:08.102016 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.101965    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:08.102016 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.101988    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:08.102016 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.102001    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ngqlp for pod openshift-network-diagnostics/network-check-target-gbxbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.102195 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.102065    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp podName:8d8f3843-88d6-45fb-9b0a-4ae823a6a064 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:12.102046481 +0000 UTC m=+11.961637838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngqlp" (UniqueName: "kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp") pod "network-check-target-gbxbc" (UID: "8d8f3843-88d6-45fb-9b0a-4ae823a6a064") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.501083 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:08.501052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:08.501246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:08.501052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:08.501246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:08.501203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:08.501246 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.501196    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:08.501406 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.501319    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:08.501406 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:08.501388    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:10.501783 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:10.501751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:10.501783 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:10.501793    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:10.502299 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:10.501762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:10.502299 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:10.501893    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:10.502299 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:10.502008    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:10.502299 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:10.502085    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:12.034103 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:12.034068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:12.034550 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:12.034350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:12.034550 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.034221    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:12.034550 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.034464    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:12.034550 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.034470    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret podName:4eccb060-886d-4453-aa0f-2160986e0827 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.034445946 +0000 UTC m=+19.894037295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret") pod "global-pull-secret-syncer-8gpqj" (UID: "4eccb060-886d-4453-aa0f-2160986e0827") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:12.034550 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.034512    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs podName:3ed98aec-15bb-4997-89c2-6606d5c377be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.034497416 +0000 UTC m=+19.894088751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs") pod "network-metrics-daemon-4q9vq" (UID: "3ed98aec-15bb-4997-89c2-6606d5c377be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:12.134731 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:12.134692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:12.134893 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.134858    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:12.134893 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.134876    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:12.134893 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.134888    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ngqlp for pod openshift-network-diagnostics/network-check-target-gbxbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:12.135133 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.134960    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp podName:8d8f3843-88d6-45fb-9b0a-4ae823a6a064 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.134943049 +0000 UTC m=+19.994534398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngqlp" (UniqueName: "kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp") pod "network-check-target-gbxbc" (UID: "8d8f3843-88d6-45fb-9b0a-4ae823a6a064") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:12.502211 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:12.502131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:12.502369 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.502241    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:12.502705 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:12.502683    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:12.502821 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.502779    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:12.502860 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:12.502836    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:12.502947 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:12.502930    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:14.501616 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:14.501570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:14.501616 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:14.501609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:14.502218 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:14.501578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:14.502218 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:14.501703    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:14.502218 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:14.501806    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:14.502218 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:14.501876    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:16.501506 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:16.501468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:16.501999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:16.501475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:16.501999 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:16.501597    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:16.501999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:16.501474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:16.501999 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:16.501680    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:16.501999 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:16.501775    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:18.501603 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:18.501567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:18.502087 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:18.501567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:18.502087 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:18.501724    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:18.502087 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:18.501573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:18.502087 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:18.501805    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:18.502087 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:18.501870    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:20.095781 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:20.095736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:20.096187 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:20.095813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:20.096187 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.095931    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:20.096187 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.095939    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:20.096187 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.095995    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret podName:4eccb060-886d-4453-aa0f-2160986e0827 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:36.09597636 +0000 UTC m=+35.955567714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret") pod "global-pull-secret-syncer-8gpqj" (UID: "4eccb060-886d-4453-aa0f-2160986e0827") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:20.096187 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.096012    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs podName:3ed98aec-15bb-4997-89c2-6606d5c377be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:36.096004597 +0000 UTC m=+35.955595931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs") pod "network-metrics-daemon-4q9vq" (UID: "3ed98aec-15bb-4997-89c2-6606d5c377be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:20.196803 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:20.196766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:20.197008 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.196980    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:20.197008 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.197007    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:20.197202 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.197019    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ngqlp for pod openshift-network-diagnostics/network-check-target-gbxbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:20.197202 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.197078    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp podName:8d8f3843-88d6-45fb-9b0a-4ae823a6a064 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:36.197060951 +0000 UTC m=+36.056652284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngqlp" (UniqueName: "kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp") pod "network-check-target-gbxbc" (UID: "8d8f3843-88d6-45fb-9b0a-4ae823a6a064") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:20.501681 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:20.501573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:20.501681 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:20.501573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:20.501924 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.501698    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:20.501924 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:20.501707    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:20.501924 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.501782    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:20.501924 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:20.501879    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:22.502442 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:22.502412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:22.502780 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:22.502536    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:22.502927 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:22.502893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:22.503412 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:22.503384    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:22.503471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:22.503438    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:22.503559 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:22.503535    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:22.575116 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:22.575083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpz2m" event={"ID":"541d06dd-ac25-4278-9164-bd5eb107c596","Type":"ContainerStarted","Data":"e1909ae2ee85b9bdccf83c9e0025954fe183f41c76423b3454a1674edd891e63"}
Sep 17 17:34:23.578906 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.578530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qx4hz" event={"ID":"ba2ba32b-2d80-4def-8d77-2899b37d9f0b","Type":"ContainerStarted","Data":"5f1aa01b3a96cdcc2447cd364406c91512910fb6f6ce4c9f8a28baa9ebf53df0"}
Sep 17 17:34:23.580116 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.580096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npxnb" event={"ID":"72e8ef0e-4a19-4a80-b7db-057ffafcfd50","Type":"ContainerStarted","Data":"50d0ada7688f936bb41e763c259c5c056bc05e2fef34915cf57a696b5d5baed6"}
Sep 17 17:34:23.582669 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.582653    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:34:23.582997 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.582978    2566 generic.go:356] "Generic (PLEG): container finished" podID="be1e098d-32d9-456d-b2e8-cf856f7f7dd9" containerID="fdd08b12b5a1c1e73ba7f31d220969261ffb6c08dff2a5273b2d2ce8cae5db66" exitCode=1
Sep 17 17:34:23.583048 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.583035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"77f936dce115f80f91071f0d792b42f86190bf262d4ac3b1c08b3e6939514090"}
Sep 17 17:34:23.583118 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.583102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"c31bac24eb4ee5055923e81a30e5438c280c4274b83b2aa31c6ef44fa9cc17eb"}
Sep 17 17:34:23.583170 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.583158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"1e43572f331e2c776d97dcde4278e5d62433f043533468b08843b0e687ee76f7"}
Sep 17 17:34:23.583220 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.583208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"9b2411f18865aeb99430e2d96a1c6e92235001941b58521225fdcc3d2387c084"}
Sep 17 17:34:23.583271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.583259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerDied","Data":"fdd08b12b5a1c1e73ba7f31d220969261ffb6c08dff2a5273b2d2ce8cae5db66"}
Sep 17 17:34:23.583322 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.583295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"ef7f6f825998998bcc112f92e815b6dca933e1529b998e209cc226835de76d28"}
Sep 17 17:34:23.584528 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.584506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp" event={"ID":"31adc1af-ab71-49d0-9a26-dae9735bb36d","Type":"ContainerStarted","Data":"14a545e3b37e1f6f8838db7fe28b19be17a2e73c0fd716e574b9e0ca8a6a6bc5"}
Sep 17 17:34:23.585902 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.585884    2566 generic.go:356] "Generic (PLEG): container finished" podID="717c1fea-058a-46b5-89c1-7b17e3b77db4" containerID="4e84e20adc1eaaccb6070a2a5cb19f18355ea8d2e5636e3a3f5501608c988083" exitCode=0
Sep 17 17:34:23.586001 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.585941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerDied","Data":"4e84e20adc1eaaccb6070a2a5cb19f18355ea8d2e5636e3a3f5501608c988083"}
Sep 17 17:34:23.587535 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.587506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5mg5t" event={"ID":"39058041-be1f-48b3-b2b2-5310ac76a7ad","Type":"ContainerStarted","Data":"64ee223217fef417d4c1e705371585356a39ca0f459e4589ce06cf50cbea52aa"}
Sep 17 17:34:23.589039 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.588940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h4cs9" event={"ID":"14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d","Type":"ContainerStarted","Data":"af9f14a19f6be64f29f6ea0d727c15aac41cc385161c1a27ed6d544e7a7f1c0e"}
Sep 17 17:34:23.604347 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.604310    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-84.ec2.internal" podStartSLOduration=20.604299293 podStartE2EDuration="20.604299293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.547727798 +0000 UTC m=+6.407319154" watchObservedRunningTime="2026-09-17 17:34:23.604299293 +0000 UTC m=+23.463890649"
Sep 17 17:34:23.625502 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.625472    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tpz2m" podStartSLOduration=11.725332445 podStartE2EDuration="21.62546379s" totalImagesPullingTime="9.900131345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.625167026 +0000 UTC m=+23.484758383" watchObservedRunningTime="2026-09-17 17:34:23.62546379 +0000 UTC m=+23.485055146"
Sep 17 17:34:23.625681 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.625664    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qx4hz" podStartSLOduration=4.400891007 podStartE2EDuration="21.625656703s" totalImagesPullingTime="17.224765696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.60399106 +0000 UTC m=+23.463582416" watchObservedRunningTime="2026-09-17 17:34:23.625656703 +0000 UTC m=+23.485248060"
Sep 17 17:34:23.640371 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.640337    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h4cs9" podStartSLOduration=4.460339776 podStartE2EDuration="21.640331224s" totalImagesPullingTime="17.179991448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.640025961 +0000 UTC m=+23.499617311" watchObservedRunningTime="2026-09-17 17:34:23.640331224 +0000 UTC m=+23.499922606"
Sep 17 17:34:23.684381 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.684347    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-npxnb" podStartSLOduration=4.504143528 podStartE2EDuration="21.684335655s" totalImagesPullingTime="17.180192127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.684089561 +0000 UTC m=+23.543680916" watchObservedRunningTime="2026-09-17 17:34:23.684335655 +0000 UTC m=+23.543927007"
Sep 17 17:34:23.705401 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.705367    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5mg5t" podStartSLOduration=4.532450794 podStartE2EDuration="21.705357606s" totalImagesPullingTime="17.172906812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.705138992 +0000 UTC m=+23.564730348" watchObservedRunningTime="2026-09-17 17:34:23.705357606 +0000 UTC m=+23.564948962"
Sep 17 17:34:23.916797 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:23.916774    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:24.437503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.437336    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:23.916792072Z","UUID":"086d64b6-e0ba-43e6-b6c2-823543dea038","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:24.439114 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.439043    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:24.439114 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.439075    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:24.501615 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.501514    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:24.501615 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.501530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:24.501838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.501514    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:24.501838 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:24.501631    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:24.501838 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:24.501734    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:24.501838 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:24.501818    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:24.593940 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.593781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lw2p5" event={"ID":"fc060665-a4d6-4f7f-90be-faaf39e1a20f","Type":"ContainerStarted","Data":"c6db516946b1f043c2a087793cf2eff59695ea620e90e62c3311fd299e30fd9d"}
Sep 17 17:34:24.596622 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.596590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp" event={"ID":"31adc1af-ab71-49d0-9a26-dae9735bb36d","Type":"ContainerStarted","Data":"5f53097dbf9f0948d10033adf2eb10aefb4d1761186a098f12fc259c24bf61be"}
Sep 17 17:34:24.614581 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:24.614534    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lw2p5" podStartSLOduration=5.437571016 podStartE2EDuration="22.614514929s" totalImagesPullingTime="17.176943913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:24.614102728 +0000 UTC m=+24.473694085" watchObservedRunningTime="2026-09-17 17:34:24.614514929 +0000 UTC m=+24.474106288"
Sep 17 17:34:25.602402 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:25.602374    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:34:25.602899 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:25.602841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"19d9944a21126c63d116f387e54a79517726e4d7b1bc7555c29b70887e68ba35"}
Sep 17 17:34:25.605192 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:25.605167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp" event={"ID":"31adc1af-ab71-49d0-9a26-dae9735bb36d","Type":"ContainerStarted","Data":"0f155041dc85ebc212f5223e634a053c8566610aa33331183d5b9b8a673912a0"}
Sep 17 17:34:25.627221 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:25.627177    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnnkp" podStartSLOduration=4.331102602 podStartE2EDuration="23.627165794s" totalImagesPullingTime="19.296063192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:25.627106224 +0000 UTC m=+25.486697580" watchObservedRunningTime="2026-09-17 17:34:25.627165794 +0000 UTC m=+25.486757150"
Sep 17 17:34:26.500853 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:26.500813    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:26.500853 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:26.500858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:26.501109 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:26.500817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:26.501109 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:26.500974    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:26.501109 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:26.501060    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:26.501243 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:26.501150    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:27.614741 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:27.614398    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:34:27.615548 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:27.615085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"085774b3ae22be1c3106eb35cd58c4b7e30150bd326c11688f94bda2d3f5c67a"}
Sep 17 17:34:27.615548 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:27.615397    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:27.615856 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:27.615602    2566 scope.go:117] "RemoveContainer" containerID="fdd08b12b5a1c1e73ba7f31d220969261ffb6c08dff2a5273b2d2ce8cae5db66"
Sep 17 17:34:27.630282 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:27.630248    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:28.347967 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.347937    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:28.348464 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.348447    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:28.501008 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.500964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:28.501147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.500964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:28.501147 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:28.501092    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:28.501147 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.500964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:28.501147 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:28.501140    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:28.501346 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:28.501207    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:28.620884 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.620819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:34:28.621576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.621215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" event={"ID":"be1e098d-32d9-456d-b2e8-cf856f7f7dd9","Type":"ContainerStarted","Data":"d1d956be21f7f2091c1445171e0151742123000d5169ff7a4f23761b707774ea"}
Sep 17 17:34:28.621576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.621451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:28.621576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.621540    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:28.623298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.623278    2566 generic.go:356] "Generic (PLEG): container finished" podID="717c1fea-058a-46b5-89c1-7b17e3b77db4" containerID="67239719525f2797908ca37470c2f18160b3865bd3bbfd626cd2ee13a4344cc9" exitCode=0
Sep 17 17:34:28.623392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.623361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerDied","Data":"67239719525f2797908ca37470c2f18160b3865bd3bbfd626cd2ee13a4344cc9"}
Sep 17 17:34:28.623516 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.623503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:28.624232 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.623943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tpz2m"
Sep 17 17:34:28.641722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.641702    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:34:28.649392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:28.649355    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx" podStartSLOduration=9.4467091 podStartE2EDuration="26.649343574s" totalImagesPullingTime="17.202634474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:28.64814591 +0000 UTC m=+28.507737267" watchObservedRunningTime="2026-09-17 17:34:28.649343574 +0000 UTC m=+28.508934930"
Sep 17 17:34:29.581035 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:29.580841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gpqj"]
Sep 17 17:34:29.581211 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:29.581155    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:29.581279 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:29.581242    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:29.585534 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:29.585504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbxbc"]
Sep 17 17:34:29.585661 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:29.585619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:29.585795 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:29.585765    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:29.586096 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:29.586075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4q9vq"]
Sep 17 17:34:29.586194 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:29.586182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:29.586302 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:29.586285    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:30.631313 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:30.631281    2566 generic.go:356] "Generic (PLEG): container finished" podID="717c1fea-058a-46b5-89c1-7b17e3b77db4" containerID="995cc6e6cd46ebca202811f62c51be29f0166c3a25e5f94541ddac2723438272" exitCode=0
Sep 17 17:34:30.631782 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:30.631348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerDied","Data":"995cc6e6cd46ebca202811f62c51be29f0166c3a25e5f94541ddac2723438272"}
Sep 17 17:34:31.501290 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:31.501260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:31.501444 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:31.501361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:31.501444 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:31.501366    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:31.501444 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:31.501395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:31.501611 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:31.501494    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:31.501611 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:31.501596    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:32.639061 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:32.639024    2566 generic.go:356] "Generic (PLEG): container finished" podID="717c1fea-058a-46b5-89c1-7b17e3b77db4" containerID="20977b3cd31b176991f5d0e210c837414bc196529c8c20c7eb1e92c02ee571d3" exitCode=0
Sep 17 17:34:32.639444 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:32.639079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerDied","Data":"20977b3cd31b176991f5d0e210c837414bc196529c8c20c7eb1e92c02ee571d3"}
Sep 17 17:34:33.501004 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:33.500972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:33.501182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:33.500972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:33.501182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:33.500972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:33.501182 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:33.501103    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q9vq" podUID="3ed98aec-15bb-4997-89c2-6606d5c377be"
Sep 17 17:34:33.501362 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:33.501182    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbxbc" podUID="8d8f3843-88d6-45fb-9b0a-4ae823a6a064"
Sep 17 17:34:33.501362 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:33.501274    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gpqj" podUID="4eccb060-886d-4453-aa0f-2160986e0827"
Sep 17 17:34:35.391155 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.391082    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-84.ec2.internal" event="NodeReady"
Sep 17 17:34:35.391554 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.391233    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:35.452412 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.452379    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dsw5m"]
Sep 17 17:34:35.476462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.476433    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j4rzw"]
Sep 17 17:34:35.476593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.476559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.479200 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.479062    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:35.479200 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.479100    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:35.479200 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.479120    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8dgbk\""
Sep 17 17:34:35.492332 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.492077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j4rzw"]
Sep 17 17:34:35.492427 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.492380    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dsw5m"]
Sep 17 17:34:35.492427 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.492142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:35.495092 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.495073    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5qv6c\""
Sep 17 17:34:35.495189 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.495133    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:35.495249 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.495220    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:35.495362 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.495343    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:35.501320 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.501298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:35.501320 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.501312    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:35.501518 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.501305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:35.503983 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.503968    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:35.504076 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.504034    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:35.504076 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.504037    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z8blr\""
Sep 17 17:34:35.504188 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.504088    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:35.504188 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.503969    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:35.504188 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.504152    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2hx6\""
Sep 17 17:34:35.606767 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.606735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.606939 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.606772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cb9384-de88-4d68-b887-c58189f35287-tmp-dir\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.606939 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.606806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a090127-3a2e-4f69-aed5-ceee9cc202d3-cert\") pod \"ingress-canary-j4rzw\" (UID: \"6a090127-3a2e-4f69-aed5-ceee9cc202d3\") " pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:35.606939 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.606880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s62q\" (UniqueName: \"kubernetes.io/projected/6a090127-3a2e-4f69-aed5-ceee9cc202d3-kube-api-access-8s62q\") pod \"ingress-canary-j4rzw\" (UID: \"6a090127-3a2e-4f69-aed5-ceee9cc202d3\") " pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:35.607101 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.606955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb9384-de88-4d68-b887-c58189f35287-config-volume\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.607101 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.607045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rlx\" (UniqueName: \"kubernetes.io/projected/57cb9384-de88-4d68-b887-c58189f35287-kube-api-access-z5rlx\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.707739 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.707661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb9384-de88-4d68-b887-c58189f35287-config-volume\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.707739 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.707726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rlx\" (UniqueName: \"kubernetes.io/projected/57cb9384-de88-4d68-b887-c58189f35287-kube-api-access-z5rlx\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.707970 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.707785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.707970 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.707806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cb9384-de88-4d68-b887-c58189f35287-tmp-dir\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.707970 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.707832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a090127-3a2e-4f69-aed5-ceee9cc202d3-cert\") pod \"ingress-canary-j4rzw\" (UID: \"6a090127-3a2e-4f69-aed5-ceee9cc202d3\") " pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:35.707970 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.707862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s62q\" (UniqueName: \"kubernetes.io/projected/6a090127-3a2e-4f69-aed5-ceee9cc202d3-kube-api-access-8s62q\") pod \"ingress-canary-j4rzw\" (UID: \"6a090127-3a2e-4f69-aed5-ceee9cc202d3\") " pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:35.707970 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:35.707906    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:35.708243 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:35.707983    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls podName:57cb9384-de88-4d68-b887-c58189f35287 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:36.207967058 +0000 UTC m=+36.067558397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls") pod "dns-default-dsw5m" (UID: "57cb9384-de88-4d68-b887-c58189f35287") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:35.708243 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.708201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cb9384-de88-4d68-b887-c58189f35287-tmp-dir\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.708402 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.708375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb9384-de88-4d68-b887-c58189f35287-config-volume\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.712546 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.712527    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a090127-3a2e-4f69-aed5-ceee9cc202d3-cert\") pod \"ingress-canary-j4rzw\" (UID: \"6a090127-3a2e-4f69-aed5-ceee9cc202d3\") " pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:35.717935 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.717900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s62q\" (UniqueName: \"kubernetes.io/projected/6a090127-3a2e-4f69-aed5-ceee9cc202d3-kube-api-access-8s62q\") pod \"ingress-canary-j4rzw\" (UID: \"6a090127-3a2e-4f69-aed5-ceee9cc202d3\") " pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:35.718036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.717941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rlx\" (UniqueName: \"kubernetes.io/projected/57cb9384-de88-4d68-b887-c58189f35287-kube-api-access-z5rlx\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:35.802278 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:35.802246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j4rzw"
Sep 17 17:34:36.111704 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.111665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:36.111889 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.111734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:34:36.111889 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:36.111848    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:34:36.112011 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:36.111950    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs podName:3ed98aec-15bb-4997-89c2-6606d5c377be nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.111901989 +0000 UTC m=+67.971493337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs") pod "network-metrics-daemon-4q9vq" (UID: "3ed98aec-15bb-4997-89c2-6606d5c377be") : secret "metrics-daemon-secret" not found
Sep 17 17:34:36.114251 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.114228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eccb060-886d-4453-aa0f-2160986e0827-original-pull-secret\") pod \"global-pull-secret-syncer-8gpqj\" (UID: \"4eccb060-886d-4453-aa0f-2160986e0827\") " pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:36.212342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.212305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:36.212527 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.212377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:36.212527 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:36.212487    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:36.212655 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:36.212560    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls podName:57cb9384-de88-4d68-b887-c58189f35287 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:37.212538968 +0000 UTC m=+37.072130323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls") pod "dns-default-dsw5m" (UID: "57cb9384-de88-4d68-b887-c58189f35287") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:36.215970 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.215944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqlp\" (UniqueName: \"kubernetes.io/projected/8d8f3843-88d6-45fb-9b0a-4ae823a6a064-kube-api-access-ngqlp\") pod \"network-check-target-gbxbc\" (UID: \"8d8f3843-88d6-45fb-9b0a-4ae823a6a064\") " pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:36.412776 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.412695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gpqj"
Sep 17 17:34:36.419964 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.419937    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:36.914202 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.914106    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gpqj"]
Sep 17 17:34:36.914750 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.914604    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j4rzw"]
Sep 17 17:34:36.915345 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:36.915322    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbxbc"]
Sep 17 17:34:36.917508 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:36.917484    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a090127_3a2e_4f69_aed5_ceee9cc202d3.slice/crio-72ae1b8942f5a422462e06092cfa164436d85e3ab264dbd2027a60d653769b8a WatchSource:0}: Error finding container 72ae1b8942f5a422462e06092cfa164436d85e3ab264dbd2027a60d653769b8a: Status 404 returned error can't find the container with id 72ae1b8942f5a422462e06092cfa164436d85e3ab264dbd2027a60d653769b8a
Sep 17 17:34:36.918247 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:36.918218    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eccb060_886d_4453_aa0f_2160986e0827.slice/crio-bf21f08d66283cd067f0b9104495dbb85807adb196c7a8f6e75141d455ff4a27 WatchSource:0}: Error finding container bf21f08d66283cd067f0b9104495dbb85807adb196c7a8f6e75141d455ff4a27: Status 404 returned error can't find the container with id bf21f08d66283cd067f0b9104495dbb85807adb196c7a8f6e75141d455ff4a27
Sep 17 17:34:36.918876 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:34:36.918853    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8f3843_88d6_45fb_9b0a_4ae823a6a064.slice/crio-b56c2c5b0d60af53c322fcf6ab3892b3418053bf5772a30cb7192e336e37d34b WatchSource:0}: Error finding container b56c2c5b0d60af53c322fcf6ab3892b3418053bf5772a30cb7192e336e37d34b: Status 404 returned error can't find the container with id b56c2c5b0d60af53c322fcf6ab3892b3418053bf5772a30cb7192e336e37d34b
Sep 17 17:34:37.219661 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:37.219597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:37.219788 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:37.219699    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:37.219788 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:37.219749    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls podName:57cb9384-de88-4d68-b887-c58189f35287 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.219735547 +0000 UTC m=+39.079326881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls") pod "dns-default-dsw5m" (UID: "57cb9384-de88-4d68-b887-c58189f35287") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:37.657559 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:37.657523    2566 generic.go:356] "Generic (PLEG): container finished" podID="717c1fea-058a-46b5-89c1-7b17e3b77db4" containerID="912607f5bca029d9881093fcec116db41a5aaa5bcdde83e01dc24fc704d31f2f" exitCode=0
Sep 17 17:34:37.658423 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:37.657609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerDied","Data":"912607f5bca029d9881093fcec116db41a5aaa5bcdde83e01dc24fc704d31f2f"}
Sep 17 17:34:37.660241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:37.660213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbxbc" event={"ID":"8d8f3843-88d6-45fb-9b0a-4ae823a6a064","Type":"ContainerStarted","Data":"b56c2c5b0d60af53c322fcf6ab3892b3418053bf5772a30cb7192e336e37d34b"}
Sep 17 17:34:37.662289 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:37.662261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gpqj" event={"ID":"4eccb060-886d-4453-aa0f-2160986e0827","Type":"ContainerStarted","Data":"bf21f08d66283cd067f0b9104495dbb85807adb196c7a8f6e75141d455ff4a27"}
Sep 17 17:34:37.664080 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:37.664033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j4rzw" event={"ID":"6a090127-3a2e-4f69-aed5-ceee9cc202d3","Type":"ContainerStarted","Data":"72ae1b8942f5a422462e06092cfa164436d85e3ab264dbd2027a60d653769b8a"}
Sep 17 17:34:38.672651 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:38.672387    2566 generic.go:356] "Generic (PLEG): container finished" podID="717c1fea-058a-46b5-89c1-7b17e3b77db4" containerID="1145a8fdaf0543ac6b4f68d2337e9694e527c3ea51a7c1c18b4d4501cec4aaf3" exitCode=0
Sep 17 17:34:38.673197 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:38.672685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerDied","Data":"1145a8fdaf0543ac6b4f68d2337e9694e527c3ea51a7c1c18b4d4501cec4aaf3"}
Sep 17 17:34:39.236571 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:39.236524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:39.236752 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:39.236694    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:39.236815 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:39.236753    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls podName:57cb9384-de88-4d68-b887-c58189f35287 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:43.236735745 +0000 UTC m=+43.096327084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls") pod "dns-default-dsw5m" (UID: "57cb9384-de88-4d68-b887-c58189f35287") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:39.681447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:39.681369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" event={"ID":"717c1fea-058a-46b5-89c1-7b17e3b77db4","Type":"ContainerStarted","Data":"8e454ab2ab58e88f982044d1c3c8f1fc775bef82e07523a74d085229867cbfa4"}
Sep 17 17:34:39.706157 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:39.706095    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dqjpg" podStartSLOduration=9.438891627 podStartE2EDuration="37.706074527s" totalImagesPullingTime="28.2671829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:39.70422941 +0000 UTC m=+39.563820765" watchObservedRunningTime="2026-09-17 17:34:39.706074527 +0000 UTC m=+39.565665886"
Sep 17 17:34:39.979208 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:39.979130    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f955969b6-hrfzg"]
Sep 17 17:34:39.995795 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:39.995767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:39.999772 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:39.999750    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f955969b6-hrfzg"]
Sep 17 17:34:40.005935 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.005897    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:34:40.006043 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.005948    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:34:40.006192 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.006173    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k4tqj\""
Sep 17 17:34:40.007938 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.007566    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:34:40.008777 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.008749    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:34:40.009312 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.009282    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:34:40.010809 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.010717    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:34:40.143190 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.143153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-default-certificate\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.143373 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.143196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.143373 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.143234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-stats-auth\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.143471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.143361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5dx\" (UniqueName: \"kubernetes.io/projected/3417e40a-cdbd-4fd3-a68f-acca57237487-kube-api-access-dk5dx\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.143471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.143420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.244032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.243948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5dx\" (UniqueName: \"kubernetes.io/projected/3417e40a-cdbd-4fd3-a68f-acca57237487-kube-api-access-dk5dx\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.244032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.244002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.244307 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.244048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-default-certificate\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.244307 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.244072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.244307 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:40.244180    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:40.244307 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:40.244216    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.744194282 +0000 UTC m=+40.603785629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:40.244307 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.244242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-stats-auth\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.244562 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:40.244320    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.744307972 +0000 UTC m=+40.603899305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : secret "router-metrics-certs-default" not found
Sep 17 17:34:40.247924 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.247884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-default-certificate\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.248043 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.247885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-stats-auth\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.253329 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.253311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5dx\" (UniqueName: \"kubernetes.io/projected/3417e40a-cdbd-4fd3-a68f-acca57237487-kube-api-access-dk5dx\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.748059 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.748023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.748537 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:40.748074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:40.748537 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:40.748195    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:40.748537 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:40.748203    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.748185546 +0000 UTC m=+41.607776885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:40.748537 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:40.748245    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.748231156 +0000 UTC m=+41.607822492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : secret "router-metrics-certs-default" not found
Sep 17 17:34:41.754997 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:41.754955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:41.755462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:41.755034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:41.755462 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:41.755173    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:41.755462 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:41.755230    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:43.75521183 +0000 UTC m=+43.614803183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : secret "router-metrics-certs-default" not found
Sep 17 17:34:41.755462 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:41.755253    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:43.755241711 +0000 UTC m=+43.614833064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:42.516375 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.516343    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8655cf6f94-vmxlq"]
Sep 17 17:34:42.531022 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.530999    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.533636 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.533605    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:34:42.533636 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.533616    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:34:42.533813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.533668    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-256jv\""
Sep 17 17:34:42.533813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.533769    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:34:42.541302 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.539796    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8655cf6f94-vmxlq"]
Sep 17 17:34:42.541302 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.541155    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:34:42.662418 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4026c36b-d825-4b7d-8c4f-c539fbb713be-ca-trust-extracted\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.662585 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-certificates\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.662585 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-bound-sa-token\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.662585 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-kube-api-access-2l9z8\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.662585 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.662585 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-trusted-ca\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.662585 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-installation-pull-secrets\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.662900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.662597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-image-registry-private-configuration\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763051 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-certificates\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-bound-sa-token\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-kube-api-access-2l9z8\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-trusted-ca\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-installation-pull-secrets\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-image-registry-private-configuration\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4026c36b-d825-4b7d-8c4f-c539fbb713be-ca-trust-extracted\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:42.763287    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:42.763310    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8655cf6f94-vmxlq: secret "image-registry-tls" not found
Sep 17 17:34:42.763397 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:42.763376    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls podName:4026c36b-d825-4b7d-8c4f-c539fbb713be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:43.263355559 +0000 UTC m=+43.122946899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls") pod "image-registry-8655cf6f94-vmxlq" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be") : secret "image-registry-tls" not found
Sep 17 17:34:42.763875 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4026c36b-d825-4b7d-8c4f-c539fbb713be-ca-trust-extracted\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.763875 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.763740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-certificates\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.764204 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.764180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-trusted-ca\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.766348 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.766327    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-image-registry-private-configuration\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.766462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.766424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-installation-pull-secrets\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.775023 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.774993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-bound-sa-token\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:42.775212 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:42.775193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-kube-api-access-2l9z8\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:43.267504 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.267461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:43.267673 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.267517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:43.267673 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.267616    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:43.267755 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.267676    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls podName:57cb9384-de88-4d68-b887-c58189f35287 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.267661159 +0000 UTC m=+51.127252497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls") pod "dns-default-dsw5m" (UID: "57cb9384-de88-4d68-b887-c58189f35287") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:43.267755 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.267698    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:43.267755 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.267710    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8655cf6f94-vmxlq: secret "image-registry-tls" not found
Sep 17 17:34:43.267755 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.267755    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls podName:4026c36b-d825-4b7d-8c4f-c539fbb713be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:44.267740064 +0000 UTC m=+44.127331400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls") pod "image-registry-8655cf6f94-vmxlq" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be") : secret "image-registry-tls" not found
Sep 17 17:34:43.580718 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.580685    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2gbv7"]
Sep 17 17:34:43.582766 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.582749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.585177 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.585148    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:34:43.585301 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.585202    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:34:43.586201 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.586183    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:34:43.586263 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.586209    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:34:43.586263 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.586193    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6zzl7\""
Sep 17 17:34:43.606808 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.606783    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2gbv7"]
Sep 17 17:34:43.693128 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.693094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbxbc" event={"ID":"8d8f3843-88d6-45fb-9b0a-4ae823a6a064","Type":"ContainerStarted","Data":"f819400abd11b791971d3757d3fec6a940696de22d86133b4db89691c64969c5"}
Sep 17 17:34:43.693292 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.693216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:34:43.694650 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.694624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gpqj" event={"ID":"4eccb060-886d-4453-aa0f-2160986e0827","Type":"ContainerStarted","Data":"ea1ea45c44cd98440eea9393921528f42fcfa9ff168dad98a50a0acbb1e83c66"}
Sep 17 17:34:43.696026 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.696004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j4rzw" event={"ID":"6a090127-3a2e-4f69-aed5-ceee9cc202d3","Type":"ContainerStarted","Data":"0da3eec171c9817814f0e15b8a1cf579d07a3a0fbaa84a56b61d8f8ee949c081"}
Sep 17 17:34:43.715199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.715157    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gbxbc" podStartSLOduration=35.884625206 podStartE2EDuration="41.715122854s" totalImagesPullingTime="5.830497648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:43.714453873 +0000 UTC m=+43.574045230" watchObservedRunningTime="2026-09-17 17:34:43.715122854 +0000 UTC m=+43.574714210"
Sep 17 17:34:43.733804 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.733763    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j4rzw" podStartSLOduration=2.901928874 podStartE2EDuration="8.733750539s" totalImagesPullingTime="5.831821665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:43.733365116 +0000 UTC m=+43.592956472" watchObservedRunningTime="2026-09-17 17:34:43.733750539 +0000 UTC m=+43.593341903"
Sep 17 17:34:43.758901 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.758851    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8gpqj" podStartSLOduration=35.916390149 podStartE2EDuration="41.758834515s" totalImagesPullingTime="5.842444366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:43.758196296 +0000 UTC m=+43.617787649" watchObservedRunningTime="2026-09-17 17:34:43.758834515 +0000 UTC m=+43.618425871"
Sep 17 17:34:43.771509 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.771483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3dca4f0-f8f3-4038-857c-a622206fa93f-data-volume\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.771527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.771556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67dwf\" (UniqueName: \"kubernetes.io/projected/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-api-access-67dwf\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.771578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3dca4f0-f8f3-4038-857c-a622206fa93f-crio-socket\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.771602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.771657    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.771636516 +0000 UTC m=+47.631227862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.771724    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.771786    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.771770663 +0000 UTC m=+47.631362014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : secret "router-metrics-certs-default" not found
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.771818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.771877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.771847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.873129 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.873045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.873129 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.873076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.873129 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.873121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3dca4f0-f8f3-4038-857c-a622206fa93f-data-volume\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.873392 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.873163    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:43.873392 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.873217    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:43.873392 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.873223    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:44.373206746 +0000 UTC m=+44.232798086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:43.873392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.873322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67dwf\" (UniqueName: \"kubernetes.io/projected/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-api-access-67dwf\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.873392 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:43.873337    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:44.373320416 +0000 UTC m=+44.232911756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:43.873612 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.873421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3dca4f0-f8f3-4038-857c-a622206fa93f-crio-socket\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.873612 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.873430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3dca4f0-f8f3-4038-857c-a622206fa93f-data-volume\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.873612 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.873588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3dca4f0-f8f3-4038-857c-a622206fa93f-crio-socket\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:43.886510 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:43.886489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67dwf\" (UniqueName: \"kubernetes.io/projected/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-api-access-67dwf\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:44.276465 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:44.276388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:44.276603 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:44.276540    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:44.276603 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:44.276556    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8655cf6f94-vmxlq: secret "image-registry-tls" not found
Sep 17 17:34:44.276679 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:44.276610    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls podName:4026c36b-d825-4b7d-8c4f-c539fbb713be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:46.276595562 +0000 UTC m=+46.136186899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls") pod "image-registry-8655cf6f94-vmxlq" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be") : secret "image-registry-tls" not found
Sep 17 17:34:44.377041 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:44.377009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:44.377041 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:44.377043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:44.377225 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:44.377134    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:44.377225 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:44.377175    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:44.377225 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:44.377197    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:45.377180373 +0000 UTC m=+45.236771726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:44.377225 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:44.377214    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:45.377205622 +0000 UTC m=+45.236796958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:45.384148 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:45.384096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:45.384148 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:45.384148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:45.384613 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:45.384225    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:45.384613 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:45.384244    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:45.384613 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:45.384291    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.384276574 +0000 UTC m=+47.243867908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:45.384613 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:45.384306    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.38429965 +0000 UTC m=+47.243890984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:45.705422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:45.705357    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npxnb_72e8ef0e-4a19-4a80-b7db-057ffafcfd50/dns-node-resolver/0.log"
Sep 17 17:34:46.291922 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:46.291878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:46.292095 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:46.292030    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:46.292095 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:46.292051    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8655cf6f94-vmxlq: secret "image-registry-tls" not found
Sep 17 17:34:46.292162 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:46.292106    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls podName:4026c36b-d825-4b7d-8c4f-c539fbb713be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.292092477 +0000 UTC m=+50.151683815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls") pod "image-registry-8655cf6f94-vmxlq" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be") : secret "image-registry-tls" not found
Sep 17 17:34:46.699722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:46.699694    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h4cs9_14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d/node-ca/0.log"
Sep 17 17:34:47.299710 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:47.299685    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j4rzw_6a090127-3a2e-4f69-aed5-ceee9cc202d3/serve-healthcheck-canary/0.log"
Sep 17 17:34:47.399291 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:47.399252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:47.399291 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:47.399288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:47.399503 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:47.399381    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:47.399503 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:47.399397    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:47.399503 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:47.399448    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.399429729 +0000 UTC m=+51.259021083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:47.399503 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:47.399468    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.399460042 +0000 UTC m=+51.259051376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:47.803207 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:47.803176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:47.803535 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:47.803292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:47.803535 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:47.803326    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:47.803535 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:47.803390    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.80337381 +0000 UTC m=+55.662965160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : secret "router-metrics-certs-default" not found
Sep 17 17:34:47.803535 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:47.803405    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.803399021 +0000 UTC m=+55.662990356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:50.319539 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:50.319499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:50.319940 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:50.319653    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:50.319940 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:50.319673    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8655cf6f94-vmxlq: secret "image-registry-tls" not found
Sep 17 17:34:50.319940 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:50.319728    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls podName:4026c36b-d825-4b7d-8c4f-c539fbb713be nodeName:}" failed. No retries permitted until 2026-09-17 17:34:58.319713535 +0000 UTC m=+58.179304868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls") pod "image-registry-8655cf6f94-vmxlq" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be") : secret "image-registry-tls" not found
Sep 17 17:34:51.325859 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:51.325811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:34:51.326333 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:51.325969    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:51.326333 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:51.326030    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls podName:57cb9384-de88-4d68-b887-c58189f35287 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:07.326015406 +0000 UTC m=+67.185606740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls") pod "dns-default-dsw5m" (UID: "57cb9384-de88-4d68-b887-c58189f35287") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:51.426447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:51.426408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:51.426447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:51.426447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:51.426703 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:51.426535    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:51.426703 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:51.426600    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:51.426703 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:51.426616    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.426594543 +0000 UTC m=+59.286185897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:51.426703 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:51.426637    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.426626727 +0000 UTC m=+59.286218061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:55.859078 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:55.859038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:55.859562 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:55.859131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:34:55.859562 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:55.859173    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:55.859562 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:55.859231    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.859217889 +0000 UTC m=+71.718809222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:55.859562 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:55.859246    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs podName:3417e40a-cdbd-4fd3-a68f-acca57237487 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.859239801 +0000 UTC m=+71.718831135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs") pod "router-default-5f955969b6-hrfzg" (UID: "3417e40a-cdbd-4fd3-a68f-acca57237487") : secret "router-metrics-certs-default" not found
Sep 17 17:34:58.373237 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:58.373191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:34:58.373615 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:58.373342    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:58.373615 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:58.373363    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8655cf6f94-vmxlq: secret "image-registry-tls" not found
Sep 17 17:34:58.373615 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:58.373420    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls podName:4026c36b-d825-4b7d-8c4f-c539fbb713be nodeName:}" failed. No retries permitted until 2026-09-17 17:35:14.373404478 +0000 UTC m=+74.232995813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls") pod "image-registry-8655cf6f94-vmxlq" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be") : secret "image-registry-tls" not found
Sep 17 17:34:59.482771 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:59.482737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:59.482771 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:59.482770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:34:59.483209 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:59.482894    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:59.483209 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:34:59.482960    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls podName:f3dca4f0-f8f3-4038-857c-a622206fa93f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:15.482948002 +0000 UTC m=+75.342539335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gbv7" (UID: "f3dca4f0-f8f3-4038-857c-a622206fa93f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:59.483283 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:34:59.483241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3dca4f0-f8f3-4038-857c-a622206fa93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:35:00.641817 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:00.641791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmbx"
Sep 17 17:35:07.340642 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:07.340606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:35:07.342885 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:07.342867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cb9384-de88-4d68-b887-c58189f35287-metrics-tls\") pod \"dns-default-dsw5m\" (UID: \"57cb9384-de88-4d68-b887-c58189f35287\") " pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:35:07.589737 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:07.589708    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8dgbk\""
Sep 17 17:35:07.597623 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:07.597568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:35:07.863143 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:07.863075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dsw5m"]
Sep 17 17:35:07.866808 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:07.866784    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cb9384_de88_4d68_b887_c58189f35287.slice/crio-16ca41042ef9b1131127e110736d9465d2dacb4d79125b06c3eedb9eb4b79bf9 WatchSource:0}: Error finding container 16ca41042ef9b1131127e110736d9465d2dacb4d79125b06c3eedb9eb4b79bf9: Status 404 returned error can't find the container with id 16ca41042ef9b1131127e110736d9465d2dacb4d79125b06c3eedb9eb4b79bf9
Sep 17 17:35:08.145872 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:08.145800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:35:08.147959 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:08.147940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed98aec-15bb-4997-89c2-6606d5c377be-metrics-certs\") pod \"network-metrics-daemon-4q9vq\" (UID: \"3ed98aec-15bb-4997-89c2-6606d5c377be\") " pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:35:08.227498 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:08.227471    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2hx6\""
Sep 17 17:35:08.235466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:08.235445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q9vq"
Sep 17 17:35:08.514768 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:08.514641    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4q9vq"]
Sep 17 17:35:08.517887 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:08.517838    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed98aec_15bb_4997_89c2_6606d5c377be.slice/crio-311effcee9296d1712e2a142244c07fd1f503be7c37fcbcc19eb6bb48227e917 WatchSource:0}: Error finding container 311effcee9296d1712e2a142244c07fd1f503be7c37fcbcc19eb6bb48227e917: Status 404 returned error can't find the container with id 311effcee9296d1712e2a142244c07fd1f503be7c37fcbcc19eb6bb48227e917
Sep 17 17:35:08.760776 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:08.760702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q9vq" event={"ID":"3ed98aec-15bb-4997-89c2-6606d5c377be","Type":"ContainerStarted","Data":"311effcee9296d1712e2a142244c07fd1f503be7c37fcbcc19eb6bb48227e917"}
Sep 17 17:35:08.764507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:08.764472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsw5m" event={"ID":"57cb9384-de88-4d68-b887-c58189f35287","Type":"ContainerStarted","Data":"16ca41042ef9b1131127e110736d9465d2dacb4d79125b06c3eedb9eb4b79bf9"}
Sep 17 17:35:09.771150 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:09.771116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsw5m" event={"ID":"57cb9384-de88-4d68-b887-c58189f35287","Type":"ContainerStarted","Data":"d042e84fd935520017a475f3074b6931176b837be0a0b1253d3d3cce03270037"}
Sep 17 17:35:09.771608 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:09.771168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsw5m" event={"ID":"57cb9384-de88-4d68-b887-c58189f35287","Type":"ContainerStarted","Data":"19c4158927d76d667b6bb86db2ad2a3f613636c92b80f79e471684784d7c8dd9"}
Sep 17 17:35:09.771608 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:09.771265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:35:09.799880 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:09.799830    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dsw5m" podStartSLOduration=33.447741698 podStartE2EDuration="34.799809956s" totalImagesPullingTime="1.352068258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:09.799135187 +0000 UTC m=+69.658726544" watchObservedRunningTime="2026-09-17 17:35:09.799809956 +0000 UTC m=+69.659401313"
Sep 17 17:35:10.776113 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:10.776080    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q9vq" event={"ID":"3ed98aec-15bb-4997-89c2-6606d5c377be","Type":"ContainerStarted","Data":"f3e111794e417243a4bb7b6ff697c6c8f9d7d96e8323b12941681ef7d5693cb4"}
Sep 17 17:35:10.776479 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:10.776130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q9vq" event={"ID":"3ed98aec-15bb-4997-89c2-6606d5c377be","Type":"ContainerStarted","Data":"807178d34ad0398838b50c18cd25840d46fe1217274360b484cac35cc7f4d14c"}
Sep 17 17:35:10.796472 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:10.796424    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4q9vq" podStartSLOduration=67.474152767 podStartE2EDuration="1m8.796412071s" totalImagesPullingTime="1.322259304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:10.794997012 +0000 UTC m=+70.654588368" watchObservedRunningTime="2026-09-17 17:35:10.796412071 +0000 UTC m=+70.656003435"
Sep 17 17:35:11.876115 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:11.876070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:11.876560 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:11.876132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:11.876805 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:11.876780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417e40a-cdbd-4fd3-a68f-acca57237487-service-ca-bundle\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:11.878413 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:11.878382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3417e40a-cdbd-4fd3-a68f-acca57237487-metrics-certs\") pod \"router-default-5f955969b6-hrfzg\" (UID: \"3417e40a-cdbd-4fd3-a68f-acca57237487\") " pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:12.109691 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:12.109664    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k4tqj\""
Sep 17 17:35:12.117668 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:12.117652    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:12.388599 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:12.388571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f955969b6-hrfzg"]
Sep 17 17:35:12.391925 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:12.391890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3417e40a_cdbd_4fd3_a68f_acca57237487.slice/crio-b401bd7f844dabbf1ebb43233300e408f16a7b4884157d0348b78a2d3302cf92 WatchSource:0}: Error finding container b401bd7f844dabbf1ebb43233300e408f16a7b4884157d0348b78a2d3302cf92: Status 404 returned error can't find the container with id b401bd7f844dabbf1ebb43233300e408f16a7b4884157d0348b78a2d3302cf92
Sep 17 17:35:12.783570 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:12.783497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f955969b6-hrfzg" event={"ID":"3417e40a-cdbd-4fd3-a68f-acca57237487","Type":"ContainerStarted","Data":"af653da1b8b4f90e6f2cab90e938d18f9dd5f0925f5843dc10bea7858e51fab2"}
Sep 17 17:35:12.783570 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:12.783540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f955969b6-hrfzg" event={"ID":"3417e40a-cdbd-4fd3-a68f-acca57237487","Type":"ContainerStarted","Data":"b401bd7f844dabbf1ebb43233300e408f16a7b4884157d0348b78a2d3302cf92"}
Sep 17 17:35:12.803024 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:12.802982    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f955969b6-hrfzg" podStartSLOduration=33.802966891 podStartE2EDuration="33.802966891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:12.802801359 +0000 UTC m=+72.662392727" watchObservedRunningTime="2026-09-17 17:35:12.802966891 +0000 UTC m=+72.662558247"
Sep 17 17:35:13.118055 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:13.117936    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:13.120384 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:13.120361    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:13.786858 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:13.786825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:13.788133 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:13.788111    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f955969b6-hrfzg"
Sep 17 17:35:14.394017 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:14.393974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:35:14.396250 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:14.396230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"image-registry-8655cf6f94-vmxlq\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") " pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:35:14.647017 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:14.646942    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-256jv\""
Sep 17 17:35:14.655082 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:14.655062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:35:14.705068 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:14.705046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gbxbc"
Sep 17 17:35:14.923426 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:14.923295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8655cf6f94-vmxlq"]
Sep 17 17:35:14.925857 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:14.925831    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4026c36b_d825_4b7d_8c4f_c539fbb713be.slice/crio-34b155b7766ec7914380c098bdc4f3415af08887ddf8843807a8a5e19673c511 WatchSource:0}: Error finding container 34b155b7766ec7914380c098bdc4f3415af08887ddf8843807a8a5e19673c511: Status 404 returned error can't find the container with id 34b155b7766ec7914380c098bdc4f3415af08887ddf8843807a8a5e19673c511
Sep 17 17:35:15.501540 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.501505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:35:15.503763 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.503740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3dca4f0-f8f3-4038-857c-a622206fa93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gbv7\" (UID: \"f3dca4f0-f8f3-4038-857c-a622206fa93f\") " pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:35:15.692815 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.692787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6zzl7\""
Sep 17 17:35:15.700854 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.700836    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2gbv7"
Sep 17 17:35:15.795967 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.795942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq" event={"ID":"4026c36b-d825-4b7d-8c4f-c539fbb713be","Type":"ContainerStarted","Data":"47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b"}
Sep 17 17:35:15.796087 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.795992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq" event={"ID":"4026c36b-d825-4b7d-8c4f-c539fbb713be","Type":"ContainerStarted","Data":"34b155b7766ec7914380c098bdc4f3415af08887ddf8843807a8a5e19673c511"}
Sep 17 17:35:15.796221 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.796201    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:35:15.816271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.816226    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq" podStartSLOduration=33.816208972 podStartE2EDuration="33.816208972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:15.814755303 +0000 UTC m=+75.674346659" watchObservedRunningTime="2026-09-17 17:35:15.816208972 +0000 UTC m=+75.675800329"
Sep 17 17:35:15.999946 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:15.999873    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2gbv7"]
Sep 17 17:35:16.002276 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:16.002252    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dca4f0_f8f3_4038_857c_a622206fa93f.slice/crio-9eb9ab0915d02f6c0ffe46dbadae14a5fc99f5d1748f4bade79394cc3b1ca1cc WatchSource:0}: Error finding container 9eb9ab0915d02f6c0ffe46dbadae14a5fc99f5d1748f4bade79394cc3b1ca1cc: Status 404 returned error can't find the container with id 9eb9ab0915d02f6c0ffe46dbadae14a5fc99f5d1748f4bade79394cc3b1ca1cc
Sep 17 17:35:16.800207 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:16.800167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gbv7" event={"ID":"f3dca4f0-f8f3-4038-857c-a622206fa93f","Type":"ContainerStarted","Data":"ca17905d0afdc29d1f0787970651d81d735593d478ba38755031ceb385500347"}
Sep 17 17:35:16.800571 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:16.800240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gbv7" event={"ID":"f3dca4f0-f8f3-4038-857c-a622206fa93f","Type":"ContainerStarted","Data":"9eb9ab0915d02f6c0ffe46dbadae14a5fc99f5d1748f4bade79394cc3b1ca1cc"}
Sep 17 17:35:17.806338 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:17.806303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gbv7" event={"ID":"f3dca4f0-f8f3-4038-857c-a622206fa93f","Type":"ContainerStarted","Data":"6c5a3cef22d1af441286925b5246e39d2275aecee55ce1827ce848dcb784d300"}
Sep 17 17:35:18.532657 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.532629    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8655cf6f94-vmxlq"]
Sep 17 17:35:18.534536 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.534513    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"]
Sep 17 17:35:18.540246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.540223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.543061 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.543034    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:35:18.543904 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.543704    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:35:18.543904 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.543755    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:35:18.543904 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.543807    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:35:18.567062 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.567040    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"]
Sep 17 17:35:18.615987 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.615965    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"]
Sep 17 17:35:18.618249 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.618232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"
Sep 17 17:35:18.621581 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.621563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-knmtx\""
Sep 17 17:35:18.623058 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.623032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:35:18.623723 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.623706    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d4cc54796-5529z"]
Sep 17 17:35:18.624793 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.624772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5vw\" (UniqueName: \"kubernetes.io/projected/65743e26-3f89-478d-b0ac-054e25fdf40b-kube-api-access-fs5vw\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.624884 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.624840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65743e26-3f89-478d-b0ac-054e25fdf40b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.624884 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.624871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65743e26-3f89-478d-b0ac-054e25fdf40b-tmp\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.625648 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.625634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.640242 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.640184    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"]
Sep 17 17:35:18.640651 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.640635    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d4cc54796-5529z"]
Sep 17 17:35:18.726115 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eef26e3-b4ac-42a5-a799-d87f15613b73-registry-certificates\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726115 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eef26e3-b4ac-42a5-a799-d87f15613b73-trusted-ca\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726328 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9kd5n\" (UID: \"5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"
Sep 17 17:35:18.726328 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-bound-sa-token\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726328 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eef26e3-b4ac-42a5-a799-d87f15613b73-installation-pull-secrets\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726328 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65743e26-3f89-478d-b0ac-054e25fdf40b-tmp\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.726471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlwr\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-kube-api-access-2rlwr\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5vw\" (UniqueName: \"kubernetes.io/projected/65743e26-3f89-478d-b0ac-054e25fdf40b-kube-api-access-fs5vw\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.726471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eef26e3-b4ac-42a5-a799-d87f15613b73-image-registry-private-configuration\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-registry-tls\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726471 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65743e26-3f89-478d-b0ac-054e25fdf40b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.726634 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eef26e3-b4ac-42a5-a799-d87f15613b73-ca-trust-extracted\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.726673 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.726629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65743e26-3f89-478d-b0ac-054e25fdf40b-tmp\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.729076 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.729052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65743e26-3f89-478d-b0ac-054e25fdf40b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.744606 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.744585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5vw\" (UniqueName: \"kubernetes.io/projected/65743e26-3f89-478d-b0ac-054e25fdf40b-kube-api-access-fs5vw\") pod \"klusterlet-addon-workmgr-5ffdbd8c89-4tr6r\" (UID: \"65743e26-3f89-478d-b0ac-054e25fdf40b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.814571 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.814543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gbv7" event={"ID":"f3dca4f0-f8f3-4038-857c-a622206fa93f","Type":"ContainerStarted","Data":"063da6d74332da206e2238756486c9135e67adbe3fef5a3aab06bb5aaec1dba9"}
Sep 17 17:35:18.827643 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-registry-tls\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.827714 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eef26e3-b4ac-42a5-a799-d87f15613b73-ca-trust-extracted\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.827714 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eef26e3-b4ac-42a5-a799-d87f15613b73-registry-certificates\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.827714 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eef26e3-b4ac-42a5-a799-d87f15613b73-trusted-ca\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.827714 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9kd5n\" (UID: \"5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"
Sep 17 17:35:18.827899 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-bound-sa-token\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.827899 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eef26e3-b4ac-42a5-a799-d87f15613b73-installation-pull-secrets\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.827899 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlwr\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-kube-api-access-2rlwr\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.827899 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.827859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eef26e3-b4ac-42a5-a799-d87f15613b73-image-registry-private-configuration\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.828128 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.828073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eef26e3-b4ac-42a5-a799-d87f15613b73-ca-trust-extracted\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.828618 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.828598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eef26e3-b4ac-42a5-a799-d87f15613b73-registry-certificates\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.828904 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.828881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eef26e3-b4ac-42a5-a799-d87f15613b73-trusted-ca\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.830320 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.830296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eef26e3-b4ac-42a5-a799-d87f15613b73-installation-pull-secrets\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.830404 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.830364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-registry-tls\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.830519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.830500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9kd5n\" (UID: \"5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"
Sep 17 17:35:18.830595 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.830575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eef26e3-b4ac-42a5-a799-d87f15613b73-image-registry-private-configuration\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.835532 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.835493    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2gbv7" podStartSLOduration=33.541548356 podStartE2EDuration="35.835482372s" totalImagesPullingTime="2.293934016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:18.834562875 +0000 UTC m=+78.694154234" watchObservedRunningTime="2026-09-17 17:35:18.835482372 +0000 UTC m=+78.695073727"
Sep 17 17:35:18.838949 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.838900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-bound-sa-token\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.839150 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.839134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlwr\" (UniqueName: \"kubernetes.io/projected/1eef26e3-b4ac-42a5-a799-d87f15613b73-kube-api-access-2rlwr\") pod \"image-registry-d4cc54796-5529z\" (UID: \"1eef26e3-b4ac-42a5-a799-d87f15613b73\") " pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:18.849969 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.849951    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:18.928928 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.928888    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"
Sep 17 17:35:18.937104 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:18.937085    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:19.132045 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.131855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"]
Sep 17 17:35:19.134304 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:19.134282    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65743e26_3f89_478d_b0ac_054e25fdf40b.slice/crio-d4371d261b22ab64279033abf6dba28299f8637c913d6c6a26adbdaa12f265ae WatchSource:0}: Error finding container d4371d261b22ab64279033abf6dba28299f8637c913d6c6a26adbdaa12f265ae: Status 404 returned error can't find the container with id d4371d261b22ab64279033abf6dba28299f8637c913d6c6a26adbdaa12f265ae
Sep 17 17:35:19.216845 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.216813    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"]
Sep 17 17:35:19.229559 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:19.229536    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0a6ea3_e191_42dd_ac7c_1e71f1b47d53.slice/crio-b3d17617dfa6538389941ee20b6dffbeab70a794661c2a8c4c3414e2442b0edd WatchSource:0}: Error finding container b3d17617dfa6538389941ee20b6dffbeab70a794661c2a8c4c3414e2442b0edd: Status 404 returned error can't find the container with id b3d17617dfa6538389941ee20b6dffbeab70a794661c2a8c4c3414e2442b0edd
Sep 17 17:35:19.242898 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.242868    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d4cc54796-5529z"]
Sep 17 17:35:19.245106 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:19.245078    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eef26e3_b4ac_42a5_a799_d87f15613b73.slice/crio-75cc4c22ad19524266448ce58390cc6176834c394cbb9575bb662d7fdd39e5c5 WatchSource:0}: Error finding container 75cc4c22ad19524266448ce58390cc6176834c394cbb9575bb662d7fdd39e5c5: Status 404 returned error can't find the container with id 75cc4c22ad19524266448ce58390cc6176834c394cbb9575bb662d7fdd39e5c5
Sep 17 17:35:19.778449 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.778416    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dsw5m"
Sep 17 17:35:19.822055 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.822017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d4cc54796-5529z" event={"ID":"1eef26e3-b4ac-42a5-a799-d87f15613b73","Type":"ContainerStarted","Data":"360edda6cac4978ddca6459d77788a26bc4c69a5b200d3ea6259178416e6f085"}
Sep 17 17:35:19.822055 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.822071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d4cc54796-5529z" event={"ID":"1eef26e3-b4ac-42a5-a799-d87f15613b73","Type":"ContainerStarted","Data":"75cc4c22ad19524266448ce58390cc6176834c394cbb9575bb662d7fdd39e5c5"}
Sep 17 17:35:19.822746 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.822726    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:19.825206 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.825180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n" event={"ID":"5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53","Type":"ContainerStarted","Data":"b3d17617dfa6538389941ee20b6dffbeab70a794661c2a8c4c3414e2442b0edd"}
Sep 17 17:35:19.827028 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.826971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r" event={"ID":"65743e26-3f89-478d-b0ac-054e25fdf40b","Type":"ContainerStarted","Data":"d4371d261b22ab64279033abf6dba28299f8637c913d6c6a26adbdaa12f265ae"}
Sep 17 17:35:19.863086 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:19.863044    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d4cc54796-5529z" podStartSLOduration=1.863028474 podStartE2EDuration="1.863028474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:19.861985072 +0000 UTC m=+79.721576425" watchObservedRunningTime="2026-09-17 17:35:19.863028474 +0000 UTC m=+79.722619861"
Sep 17 17:35:20.833126 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:20.833086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n" event={"ID":"5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53","Type":"ContainerStarted","Data":"ffa35ffd7b301026b63f3721b1a66812aad7234f036f07cf94c24ab868914ef7"}
Sep 17 17:35:20.833559 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:20.833341    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"
Sep 17 17:35:20.839521 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:20.839482    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n"
Sep 17 17:35:20.853468 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:20.853418    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9kd5n" podStartSLOduration=1.694290714 podStartE2EDuration="2.853398357s" totalImagesPullingTime="1.159107643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:20.852430372 +0000 UTC m=+80.712021730" watchObservedRunningTime="2026-09-17 17:35:20.853398357 +0000 UTC m=+80.712989714"
Sep 17 17:35:23.844352 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:23.844318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r" event={"ID":"65743e26-3f89-478d-b0ac-054e25fdf40b","Type":"ContainerStarted","Data":"04aba48e0ad3643aa1c89a3f30a299aca4607d4702ccb5a667e8064ae09c715d"}
Sep 17 17:35:23.844762 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:23.844508    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:23.846234 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:23.846211    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r"
Sep 17 17:35:23.861256 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:23.861190    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ffdbd8c89-4tr6r" podStartSLOduration=1.6003240239999998 podStartE2EDuration="5.861177008s" totalImagesPullingTime="4.260852984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:23.860066275 +0000 UTC m=+83.719657628" watchObservedRunningTime="2026-09-17 17:35:23.861177008 +0000 UTC m=+83.720768364"
Sep 17 17:35:26.171448 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.171411    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ks9d5"]
Sep 17 17:35:26.174310 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.174288    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.180420 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.180399    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twsbr\""
Sep 17 17:35:26.180515 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.180399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:35:26.181447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.181431    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:35:26.181519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.181441    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:35:26.181519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.181457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:35:26.181519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.181463    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:35:26.181669 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.181431    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:35:26.286311 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-sys\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a5b31a0-b429-4ccc-8961-f246207f6791-metrics-client-ca\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-root\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-tls\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-textfile\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm5q\" (UniqueName: \"kubernetes.io/projected/4a5b31a0-b429-4ccc-8961-f246207f6791-kube-api-access-nhm5q\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-wtmp\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-accelerators-collector-config\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.286593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.286560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.386827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.386796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-textfile\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.386827 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.386828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm5q\" (UniqueName: \"kubernetes.io/projected/4a5b31a0-b429-4ccc-8961-f246207f6791-kube-api-access-nhm5q\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.386844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-wtmp\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.386862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-accelerators-collector-config\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.386882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.386940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-sys\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.386992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-sys\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387243 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-wtmp\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387243 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a5b31a0-b429-4ccc-8961-f246207f6791-metrics-client-ca\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387243 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-root\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387243 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-tls\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387243 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a5b31a0-b429-4ccc-8961-f246207f6791-root\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387243 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:35:26.387212    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:35:26.387438 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:35:26.387290    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-tls podName:4a5b31a0-b429-4ccc-8961-f246207f6791 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:26.887267625 +0000 UTC m=+86.746858968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-tls") pod "node-exporter-ks9d5" (UID: "4a5b31a0-b429-4ccc-8961-f246207f6791") : secret "node-exporter-tls" not found
Sep 17 17:35:26.387547 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-accelerators-collector-config\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387585 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-textfile\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.387634 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.387618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a5b31a0-b429-4ccc-8961-f246207f6791-metrics-client-ca\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.389363 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.389340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.412370 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.412345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm5q\" (UniqueName: \"kubernetes.io/projected/4a5b31a0-b429-4ccc-8961-f246207f6791-kube-api-access-nhm5q\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.891111 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.891078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-tls\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:26.893352 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:26.893322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a5b31a0-b429-4ccc-8961-f246207f6791-node-exporter-tls\") pod \"node-exporter-ks9d5\" (UID: \"4a5b31a0-b429-4ccc-8961-f246207f6791\") " pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:27.083021 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.082986    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ks9d5"
Sep 17 17:35:27.094700 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:27.094668    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5b31a0_b429_4ccc_8961_f246207f6791.slice/crio-0c981a40c9d7989702b45b05590644492bc89f82484edaf350ddec79dd5d16f6 WatchSource:0}: Error finding container 0c981a40c9d7989702b45b05590644492bc89f82484edaf350ddec79dd5d16f6: Status 404 returned error can't find the container with id 0c981a40c9d7989702b45b05590644492bc89f82484edaf350ddec79dd5d16f6
Sep 17 17:35:27.283339 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.283263    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:35:27.286501 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.286479    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.289756 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.289724    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:35:27.289850 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.289735    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:35:27.290081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290064    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:35:27.290299 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:35:27.290398 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:35:27.290398 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290332    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:35:27.290551 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290537    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vqbj4\""
Sep 17 17:35:27.290590 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290558    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:35:27.290621 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290606    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:35:27.290802 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.290788    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:35:27.303800 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.303743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:35:27.393889 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.393851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394054 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.393892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394054 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.393937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394054 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.393971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394054 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.393998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394054 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jfpp\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-kube-api-access-2jfpp\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-web-config\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-config-out\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.394258 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.394228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495044 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495044 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jfpp\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-kube-api-access-2jfpp\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-web-config\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-config-out\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495557 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495557 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495557 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495557 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495557 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:35:27.495472    2566 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 17:35:27.495557 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.495557 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:35:27.495527    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls podName:ff56b308-d378-4bd2-9db2-4264b8053e41 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:27.995509382 +0000 UTC m=+87.855100719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41") : secret "alertmanager-main-tls" not found
Sep 17 17:35:27.495883 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.495779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.496149 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.496123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498395 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-web-config\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498485 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498485 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498597 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498735 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498797 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498797 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-config-out\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.498797 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.498787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.505019 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.505001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jfpp\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-kube-api-access-2jfpp\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:27.860211 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:27.860167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks9d5" event={"ID":"4a5b31a0-b429-4ccc-8961-f246207f6791","Type":"ContainerStarted","Data":"0c981a40c9d7989702b45b05590644492bc89f82484edaf350ddec79dd5d16f6"}
Sep 17 17:35:28.000768 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:28.000735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:28.002901 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:28.002868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:28.196654 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:28.196588    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:35:28.475792 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:28.475765    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:35:28.477288 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:28.477260    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff56b308_d378_4bd2_9db2_4264b8053e41.slice/crio-4c84454e2730bb0efa5392c0254720f57454c3a8bf96e9ea845e72cfa53f2a50 WatchSource:0}: Error finding container 4c84454e2730bb0efa5392c0254720f57454c3a8bf96e9ea845e72cfa53f2a50: Status 404 returned error can't find the container with id 4c84454e2730bb0efa5392c0254720f57454c3a8bf96e9ea845e72cfa53f2a50
Sep 17 17:35:28.865456 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:28.865422    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a5b31a0-b429-4ccc-8961-f246207f6791" containerID="4facece3b212a5e0047ca33335c3fba36a83d682b6674ce8c1408c0b53f98e23" exitCode=0
Sep 17 17:35:28.865621 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:28.865495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks9d5" event={"ID":"4a5b31a0-b429-4ccc-8961-f246207f6791","Type":"ContainerDied","Data":"4facece3b212a5e0047ca33335c3fba36a83d682b6674ce8c1408c0b53f98e23"}
Sep 17 17:35:28.866764 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:28.866746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerStarted","Data":"4c84454e2730bb0efa5392c0254720f57454c3a8bf96e9ea845e72cfa53f2a50"}
Sep 17 17:35:29.872107 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:29.872016    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerID="2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb" exitCode=0
Sep 17 17:35:29.872521 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:29.872103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb"}
Sep 17 17:35:29.874518 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:29.874494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks9d5" event={"ID":"4a5b31a0-b429-4ccc-8961-f246207f6791","Type":"ContainerStarted","Data":"f3f80ba60526a8d9cc88fb5170623a971dfee01eb31331f7b2f3cb576d04b2cc"}
Sep 17 17:35:29.874613 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:29.874537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks9d5" event={"ID":"4a5b31a0-b429-4ccc-8961-f246207f6791","Type":"ContainerStarted","Data":"116792f24249dd5db61aaff9b03450cf400bbf9b60fa5cae8830138bbf232818"}
Sep 17 17:35:29.943754 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:29.943707    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ks9d5" podStartSLOduration=3.009742357 podStartE2EDuration="3.943691413s" totalImagesPullingTime="873.01133ms" totalInitContainerRuntime="60.937726ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:29.942673714 +0000 UTC m=+89.802265070" watchObservedRunningTime="2026-09-17 17:35:29.943691413 +0000 UTC m=+89.803282789"
Sep 17 17:35:30.906126 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:30.906085    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"]
Sep 17 17:35:30.909152 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:30.909128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:30.911425 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:30.911398    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 17:35:30.911425 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:30.911420    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dhllc\""
Sep 17 17:35:30.922479 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:30.921598    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"]
Sep 17 17:35:31.026083 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.026054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bf563f2-03ff-4b66-982a-b40bf7d4501e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-t82j5\" (UID: \"0bf563f2-03ff-4b66-982a-b40bf7d4501e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.026262 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.026127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncf8s\" (UniqueName: \"kubernetes.io/projected/0bf563f2-03ff-4b66-982a-b40bf7d4501e-kube-api-access-ncf8s\") pod \"monitoring-plugin-5984459ff8-t82j5\" (UID: \"0bf563f2-03ff-4b66-982a-b40bf7d4501e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.127563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.127519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bf563f2-03ff-4b66-982a-b40bf7d4501e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-t82j5\" (UID: \"0bf563f2-03ff-4b66-982a-b40bf7d4501e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.127744 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.127625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncf8s\" (UniqueName: \"kubernetes.io/projected/0bf563f2-03ff-4b66-982a-b40bf7d4501e-kube-api-access-ncf8s\") pod \"monitoring-plugin-5984459ff8-t82j5\" (UID: \"0bf563f2-03ff-4b66-982a-b40bf7d4501e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.127744 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:35:31.127652    2566 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 17:35:31.127744 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:35:31.127721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bf563f2-03ff-4b66-982a-b40bf7d4501e-monitoring-plugin-cert podName:0bf563f2-03ff-4b66-982a-b40bf7d4501e nodeName:}" failed. No retries permitted until 2026-09-17 17:35:31.627701088 +0000 UTC m=+91.487292437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0bf563f2-03ff-4b66-982a-b40bf7d4501e-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-t82j5" (UID: "0bf563f2-03ff-4b66-982a-b40bf7d4501e") : secret "monitoring-plugin-cert" not found
Sep 17 17:35:31.137390 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.137363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncf8s\" (UniqueName: \"kubernetes.io/projected/0bf563f2-03ff-4b66-982a-b40bf7d4501e-kube-api-access-ncf8s\") pod \"monitoring-plugin-5984459ff8-t82j5\" (UID: \"0bf563f2-03ff-4b66-982a-b40bf7d4501e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.631176 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.631147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bf563f2-03ff-4b66-982a-b40bf7d4501e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-t82j5\" (UID: \"0bf563f2-03ff-4b66-982a-b40bf7d4501e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.633363 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.633338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bf563f2-03ff-4b66-982a-b40bf7d4501e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-t82j5\" (UID: \"0bf563f2-03ff-4b66-982a-b40bf7d4501e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.825519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.825495    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:31.887700 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.887519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerStarted","Data":"dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776"}
Sep 17 17:35:31.887700 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.887584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerStarted","Data":"35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507"}
Sep 17 17:35:31.887700 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.887616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerStarted","Data":"6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5"}
Sep 17 17:35:31.887700 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.887640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerStarted","Data":"bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467"}
Sep 17 17:35:31.887700 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:31.887669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerStarted","Data":"ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a"}
Sep 17 17:35:32.093123 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.093051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"]
Sep 17 17:35:32.095710 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:32.095677    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf563f2_03ff_4b66_982a_b40bf7d4501e.slice/crio-22e1c5bf4b1c1d931b722ebc5f08843d18e75405f8708b9e750f4646e70e0e4e WatchSource:0}: Error finding container 22e1c5bf4b1c1d931b722ebc5f08843d18e75405f8708b9e750f4646e70e0e4e: Status 404 returned error can't find the container with id 22e1c5bf4b1c1d931b722ebc5f08843d18e75405f8708b9e750f4646e70e0e4e
Sep 17 17:35:32.399178 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.399150    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:35:32.401659 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.401644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.404173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.404138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:35:32.404380 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.404368    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:35:32.404578 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.404563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:35:32.404760 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.404747    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ll73hqjiajmk\""
Sep 17 17:35:32.405266 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.405247    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:35:32.405355 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.405279    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:35:32.405355 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.405345    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:35:32.405636 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.405620    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:35:32.405731 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.405621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:35:32.405921 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.405895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:35:32.406303 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.406095    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:35:32.406303 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.406121    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:35:32.406303 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.406134    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:35:32.406303 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.406160    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:35:32.406303 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.406210    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zscrq\""
Sep 17 17:35:32.409087 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.409069    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:35:32.411490 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.411469    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:35:32.418945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.418924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:35:32.538072 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-web-config\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-config-out\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-config\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538239 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crnm9\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-kube-api-access-crnm9\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538862 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538862 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.538862 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.538626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639037 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.638959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-config\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639037 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.638995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639037 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639013    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crnm9\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-kube-api-access-crnm9\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639442 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639442 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639442 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639442 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639442 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639442 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-web-config\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-config-out\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.639709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.640081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.639988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.640081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.640022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.640180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.640144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.640327 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.640302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.640965 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.640939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.642240 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.642214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.642365 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.642347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.642401 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.642372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-config\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.642769 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.642745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.643181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.643076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.643497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.643470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.643588 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.643549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.643648 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.643623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.644005 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.643870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.644005 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.643899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.644954 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.644933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.645119 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.645099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-web-config\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.645392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.645377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-config-out\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.645767 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.645741    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.648290 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.648262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crnm9\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-kube-api-access-crnm9\") pod \"prometheus-k8s-0\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.712255 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.712216    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:32.893365 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.893280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5" event={"ID":"0bf563f2-03ff-4b66-982a-b40bf7d4501e","Type":"ContainerStarted","Data":"22e1c5bf4b1c1d931b722ebc5f08843d18e75405f8708b9e750f4646e70e0e4e"}
Sep 17 17:35:32.897009 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.896979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerStarted","Data":"fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9"}
Sep 17 17:35:32.929221 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.929173    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.663731838 podStartE2EDuration="5.929156277s" totalImagesPullingTime="3.249925662s" totalInitContainerRuntime="15.498777ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:32.926949803 +0000 UTC m=+92.786541159" watchObservedRunningTime="2026-09-17 17:35:32.929156277 +0000 UTC m=+92.788747636"
Sep 17 17:35:32.997584 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:32.997552    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:35:33.000463 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:35:33.000430    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514016aa_8741_41fb_b156_cd480ae00172.slice/crio-7ddb0768da450ec97917da45231c309de3e6f262f63752764cf67c5bae542be5 WatchSource:0}: Error finding container 7ddb0768da450ec97917da45231c309de3e6f262f63752764cf67c5bae542be5: Status 404 returned error can't find the container with id 7ddb0768da450ec97917da45231c309de3e6f262f63752764cf67c5bae542be5
Sep 17 17:35:33.902426 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:33.902385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5" event={"ID":"0bf563f2-03ff-4b66-982a-b40bf7d4501e","Type":"ContainerStarted","Data":"029a97ae1feadad58519b54552b2aa1a6a86e53f718887bea4301639a203971e"}
Sep 17 17:35:33.902874 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:33.902453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:33.904068 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:33.904043    2566 generic.go:356] "Generic (PLEG): container finished" podID="514016aa-8741-41fb-b156-cd480ae00172" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23" exitCode=0
Sep 17 17:35:33.904175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:33.904129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"}
Sep 17 17:35:33.904211 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:33.904185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerStarted","Data":"7ddb0768da450ec97917da45231c309de3e6f262f63752764cf67c5bae542be5"}
Sep 17 17:35:33.907817 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:33.907797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5"
Sep 17 17:35:33.919197 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:33.919162    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-t82j5" podStartSLOduration=2.5687458850000002 podStartE2EDuration="3.919148889s" totalImagesPullingTime="1.350403004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:33.918082524 +0000 UTC m=+93.777673879" watchObservedRunningTime="2026-09-17 17:35:33.919148889 +0000 UTC m=+93.778740245"
Sep 17 17:35:37.924464 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:37.924424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerStarted","Data":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"}
Sep 17 17:35:37.924839 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:37.924514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerStarted","Data":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"}
Sep 17 17:35:38.539425 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:38.539396    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:35:39.935706 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:39.935671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerStarted","Data":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"}
Sep 17 17:35:39.936068 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:39.935760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerStarted","Data":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"}
Sep 17 17:35:39.936068 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:39.935787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerStarted","Data":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"}
Sep 17 17:35:39.936068 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:39.935815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerStarted","Data":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"}
Sep 17 17:35:39.967697 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:39.967647    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.92673726 podStartE2EDuration="7.967630459s" totalImagesPullingTime="5.023326201s" totalInitContainerRuntime="17.566998ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:39.965258868 +0000 UTC m=+99.824850225" watchObservedRunningTime="2026-09-17 17:35:39.967630459 +0000 UTC m=+99.827221814"
Sep 17 17:35:41.840711 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:41.840682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d4cc54796-5529z"
Sep 17 17:35:42.712519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:42.712490    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:43.552758 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.552710    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq" podUID="4026c36b-d825-4b7d-8c4f-c539fbb713be" containerName="registry" containerID="cri-o://47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b" gracePeriod=30
Sep 17 17:35:43.790813 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.790793    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:35:43.937564 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-certificates\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.937564 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937525    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-kube-api-access-2l9z8\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.937564 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937544    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-bound-sa-token\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.937780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-image-registry-private-configuration\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.937780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-installation-pull-secrets\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.937780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937652    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.937780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937674    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4026c36b-d825-4b7d-8c4f-c539fbb713be-ca-trust-extracted\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.937780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-trusted-ca\") pod \"4026c36b-d825-4b7d-8c4f-c539fbb713be\" (UID: \"4026c36b-d825-4b7d-8c4f-c539fbb713be\") "
Sep 17 17:35:43.938067 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.937949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:35:43.938381 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.938343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:35:43.940151 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.940101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:35:43.940151 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.940130    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:35:43.940340 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.940318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-kube-api-access-2l9z8" (OuterVolumeSpecName: "kube-api-access-2l9z8") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "kube-api-access-2l9z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:35:43.940381 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.940333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:35:43.940381 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.940357    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:35:43.946368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.946347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4026c36b-d825-4b7d-8c4f-c539fbb713be-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4026c36b-d825-4b7d-8c4f-c539fbb713be" (UID: "4026c36b-d825-4b7d-8c4f-c539fbb713be"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:35:43.952986 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.952955    2566 generic.go:356] "Generic (PLEG): container finished" podID="4026c36b-d825-4b7d-8c4f-c539fbb713be" containerID="47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b" exitCode=0
Sep 17 17:35:43.953091 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.953044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq" event={"ID":"4026c36b-d825-4b7d-8c4f-c539fbb713be","Type":"ContainerDied","Data":"47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b"}
Sep 17 17:35:43.953091 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.953083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq" event={"ID":"4026c36b-d825-4b7d-8c4f-c539fbb713be","Type":"ContainerDied","Data":"34b155b7766ec7914380c098bdc4f3415af08887ddf8843807a8a5e19673c511"}
Sep 17 17:35:43.953091 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.953084    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8655cf6f94-vmxlq"
Sep 17 17:35:43.953245 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.953113    2566 scope.go:117] "RemoveContainer" containerID="47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b"
Sep 17 17:35:43.961562 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.961544    2566 scope.go:117] "RemoveContainer" containerID="47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b"
Sep 17 17:35:43.961810 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:35:43.961790    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b\": container with ID starting with 47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b not found: ID does not exist" containerID="47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b"
Sep 17 17:35:43.961944 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.961825    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b"} err="failed to get container status \"47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b\": rpc error: code = NotFound desc = could not find container \"47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b\": container with ID starting with 47d58f7969e1b1f4c25fad786e7e894750a195d5d502d7a91cdaee3685e2b07b not found: ID does not exist"
Sep 17 17:35:43.980394 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:43.980371    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8655cf6f94-vmxlq"]
Sep 17 17:35:44.009957 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.009932    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8655cf6f94-vmxlq"]
Sep 17 17:35:44.038805 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038783    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-image-registry-private-configuration\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.038900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038803    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4026c36b-d825-4b7d-8c4f-c539fbb713be-installation-pull-secrets\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.038900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038820    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-tls\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.038900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038833    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4026c36b-d825-4b7d-8c4f-c539fbb713be-ca-trust-extracted\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.038900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038845    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-trusted-ca\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.038900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038858    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4026c36b-d825-4b7d-8c4f-c539fbb713be-registry-certificates\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.038900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038871    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-kube-api-access-2l9z8\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.038900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.038884    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4026c36b-d825-4b7d-8c4f-c539fbb713be-bound-sa-token\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:35:44.505612 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:35:44.505576    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4026c36b-d825-4b7d-8c4f-c539fbb713be" path="/var/lib/kubelet/pods/4026c36b-d825-4b7d-8c4f-c539fbb713be/volumes"
Sep 17 17:36:32.712816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:32.712774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:32.728221 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:32.728199    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:33.140240 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:33.140210    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:46.476529 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:46.476452    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:46.477149 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:46.476761    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="alertmanager" containerID="cri-o://ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a" gracePeriod=120
Sep 17 17:36:46.477149 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:46.476837    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-metric" containerID="cri-o://dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776" gracePeriod=120
Sep 17 17:36:46.477149 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:46.476854    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-web" containerID="cri-o://6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5" gracePeriod=120
Sep 17 17:36:46.477149 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:46.476889    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy" containerID="cri-o://35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507" gracePeriod=120
Sep 17 17:36:46.477149 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:46.476863    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="config-reloader" containerID="cri-o://bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467" gracePeriod=120
Sep 17 17:36:46.477149 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:46.477031    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="prom-label-proxy" containerID="cri-o://fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9" gracePeriod=120
Sep 17 17:36:47.178698 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178663    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerID="fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9" exitCode=0
Sep 17 17:36:47.178698 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178688    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerID="dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776" exitCode=0
Sep 17 17:36:47.178698 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178695    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerID="35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507" exitCode=0
Sep 17 17:36:47.178698 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178702    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerID="bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467" exitCode=0
Sep 17 17:36:47.178698 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178707    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerID="ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a" exitCode=0
Sep 17 17:36:47.179089 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9"}
Sep 17 17:36:47.179089 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776"}
Sep 17 17:36:47.179089 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507"}
Sep 17 17:36:47.179089 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467"}
Sep 17 17:36:47.179089 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.178947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a"}
Sep 17 17:36:47.718740 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.718719    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:47.826501 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826421    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826501 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826501 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-main-db\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826501 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826493    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-web-config\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-config-volume\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-tls-assets\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jfpp\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-kube-api-access-2jfpp\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-metrics-client-ca\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826673    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-trusted-ca-bundle\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826717    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-cluster-tls-config\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-config-out\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.826816 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.827253 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-web\") pod \"ff56b308-d378-4bd2-9db2-4264b8053e41\" (UID: \"ff56b308-d378-4bd2-9db2-4264b8053e41\") "
Sep 17 17:36:47.827253 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.826827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:47.827253 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.827068    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-main-db\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.827253 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.827206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:47.827621 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.827596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:47.829206 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.829174    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-kube-api-access-2jfpp" (OuterVolumeSpecName: "kube-api-access-2jfpp") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "kube-api-access-2jfpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:47.829311 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.829213    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:47.829944 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.829882    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-config-volume" (OuterVolumeSpecName: "config-volume") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:47.830051 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.830031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:47.830121 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.830105    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-config-out" (OuterVolumeSpecName: "config-out") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:47.830204 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.830182    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:47.830437 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.830420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:47.831264 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.831231    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:47.834331 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.834307    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:47.839180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.839153    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-web-config" (OuterVolumeSpecName: "web-config") pod "ff56b308-d378-4bd2-9db2-4264b8053e41" (UID: "ff56b308-d378-4bd2-9db2-4264b8053e41"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:47.927601 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927579    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-tls-assets\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927601 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927599    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jfpp\" (UniqueName: \"kubernetes.io/projected/ff56b308-d378-4bd2-9db2-4264b8053e41-kube-api-access-2jfpp\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927608    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-metrics-client-ca\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927616    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff56b308-d378-4bd2-9db2-4264b8053e41-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927625    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-cluster-tls-config\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927634    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff56b308-d378-4bd2-9db2-4264b8053e41-config-out\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927643    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927652    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927660    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927668    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-secret-alertmanager-main-tls\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927676    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-web-config\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:47.927715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:47.927684    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff56b308-d378-4bd2-9db2-4264b8053e41-config-volume\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:48.186011 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.185929    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerID="6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5" exitCode=0
Sep 17 17:36:48.186140 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.186004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5"}
Sep 17 17:36:48.186140 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.186023    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.186140 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.186103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff56b308-d378-4bd2-9db2-4264b8053e41","Type":"ContainerDied","Data":"4c84454e2730bb0efa5392c0254720f57454c3a8bf96e9ea845e72cfa53f2a50"}
Sep 17 17:36:48.186140 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.186139    2566 scope.go:117] "RemoveContainer" containerID="fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9"
Sep 17 17:36:48.193726 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.193703    2566 scope.go:117] "RemoveContainer" containerID="dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776"
Sep 17 17:36:48.201037 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.201022    2566 scope.go:117] "RemoveContainer" containerID="35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507"
Sep 17 17:36:48.207248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.207231    2566 scope.go:117] "RemoveContainer" containerID="6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5"
Sep 17 17:36:48.210699 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.210679    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:48.214009 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.213990    2566 scope.go:117] "RemoveContainer" containerID="bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467"
Sep 17 17:36:48.218060 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.218040    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:48.220946 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.220934    2566 scope.go:117] "RemoveContainer" containerID="ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a"
Sep 17 17:36:48.227272 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.227255    2566 scope.go:117] "RemoveContainer" containerID="2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb"
Sep 17 17:36:48.233529 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.233516    2566 scope.go:117] "RemoveContainer" containerID="fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9"
Sep 17 17:36:48.233787 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:48.233770    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9\": container with ID starting with fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9 not found: ID does not exist" containerID="fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9"
Sep 17 17:36:48.233826 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.233795    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9"} err="failed to get container status \"fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9\": rpc error: code = NotFound desc = could not find container \"fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9\": container with ID starting with fa02530aad20f1d0dc5b08d6ad923917755488ca35dc93e5544a9e13756315c9 not found: ID does not exist"
Sep 17 17:36:48.233826 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.233810    2566 scope.go:117] "RemoveContainer" containerID="dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776"
Sep 17 17:36:48.234059 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:48.234045    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776\": container with ID starting with dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776 not found: ID does not exist" containerID="dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776"
Sep 17 17:36:48.234104 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234064    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776"} err="failed to get container status \"dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776\": rpc error: code = NotFound desc = could not find container \"dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776\": container with ID starting with dda636ea30aa7aecf4c916a233f2c92e6605cf42584987ff25e1c1cb29e62776 not found: ID does not exist"
Sep 17 17:36:48.234104 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234076    2566 scope.go:117] "RemoveContainer" containerID="35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507"
Sep 17 17:36:48.234302 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:48.234284    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507\": container with ID starting with 35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507 not found: ID does not exist" containerID="35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507"
Sep 17 17:36:48.234336 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234309    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507"} err="failed to get container status \"35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507\": rpc error: code = NotFound desc = could not find container \"35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507\": container with ID starting with 35a5f830f1f5665271c5a972ae836df5c7dacf989ed82614eafebb6723242507 not found: ID does not exist"
Sep 17 17:36:48.234336 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234327    2566 scope.go:117] "RemoveContainer" containerID="6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5"
Sep 17 17:36:48.234574 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:48.234553    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5\": container with ID starting with 6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5 not found: ID does not exist" containerID="6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5"
Sep 17 17:36:48.234655 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234574    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5"} err="failed to get container status \"6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5\": rpc error: code = NotFound desc = could not find container \"6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5\": container with ID starting with 6ae3b120a974f655900dd766265359fe191746b93bbb1a12fa5dc3d4f47a32c5 not found: ID does not exist"
Sep 17 17:36:48.234655 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234587    2566 scope.go:117] "RemoveContainer" containerID="bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467"
Sep 17 17:36:48.234796 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:48.234782    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467\": container with ID starting with bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467 not found: ID does not exist" containerID="bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467"
Sep 17 17:36:48.234839 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234799    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467"} err="failed to get container status \"bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467\": rpc error: code = NotFound desc = could not find container \"bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467\": container with ID starting with bc5f49e3b362d6fced9e0ff1d22fce410f85853ca9577575baabcd4fce671467 not found: ID does not exist"
Sep 17 17:36:48.234839 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.234815    2566 scope.go:117] "RemoveContainer" containerID="ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a"
Sep 17 17:36:48.235033 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:48.235016    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a\": container with ID starting with ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a not found: ID does not exist" containerID="ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a"
Sep 17 17:36:48.235102 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.235039    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a"} err="failed to get container status \"ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a\": rpc error: code = NotFound desc = could not find container \"ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a\": container with ID starting with ab5618fbe028319cf7818ec85286fd6c187d0ae6851dc3a5551b9ca17e7bf98a not found: ID does not exist"
Sep 17 17:36:48.235102 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.235057    2566 scope.go:117] "RemoveContainer" containerID="2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb"
Sep 17 17:36:48.235466 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:48.235442    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb\": container with ID starting with 2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb not found: ID does not exist" containerID="2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb"
Sep 17 17:36:48.235516 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.235468    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb"} err="failed to get container status \"2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb\": rpc error: code = NotFound desc = could not find container \"2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb\": container with ID starting with 2b28247a741a7bf859596b69d65a0ac1aa8fcdf1b31709a883a5af515de39beb not found: ID does not exist"
Sep 17 17:36:48.245710 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245687    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:48.245959 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245946    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy"
Sep 17 17:36:48.245999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245961    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy"
Sep 17 17:36:48.245999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245970    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-metric"
Sep 17 17:36:48.245999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245977    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-metric"
Sep 17 17:36:48.245999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245983    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4026c36b-d825-4b7d-8c4f-c539fbb713be" containerName="registry"
Sep 17 17:36:48.245999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245989    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4026c36b-d825-4b7d-8c4f-c539fbb713be" containerName="registry"
Sep 17 17:36:48.245999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.245999    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="init-config-reloader"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246008    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="init-config-reloader"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246016    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="alertmanager"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246021    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="alertmanager"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246027    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="prom-label-proxy"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246032    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="prom-label-proxy"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246041    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-web"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246047    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-web"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246053    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="config-reloader"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246058    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="config-reloader"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246110    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-web"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246117    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="prom-label-proxy"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246123    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4026c36b-d825-4b7d-8c4f-c539fbb713be" containerName="registry"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246128    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="config-reloader"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246135    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="alertmanager"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246140    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy"
Sep 17 17:36:48.246173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.246146    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" containerName="kube-rbac-proxy-metric"
Sep 17 17:36:48.250956 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.250942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.253375 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.253354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:36:48.253468 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.253441    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:36:48.253531 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.253507    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:36:48.253594 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.253582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:36:48.253653 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.253590    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:36:48.253653 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.253635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:36:48.254016 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.254000    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:36:48.254081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.254046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:36:48.254262 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.254238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vqbj4\""
Sep 17 17:36:48.259233 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.259213    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:36:48.264387 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.264367    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:48.331470 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331470 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331630 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncrh\" (UniqueName: \"kubernetes.io/projected/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-kube-api-access-jncrh\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331630 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331630 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331630 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331630 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331781 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-config-out\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331781 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-web-config\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331781 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331781 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331781 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.331955 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.331779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.432896 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.432868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.432896 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.432896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433097 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.432951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-config-out\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433097 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.432969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-web-config\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433097 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.432991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433097 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433097 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433097 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433378 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433378 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433378 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncrh\" (UniqueName: \"kubernetes.io/projected/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-kube-api-access-jncrh\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433378 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.433378 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.434169 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.433797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.434292 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.434193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-config-out\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-web-config\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436410 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436674 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.436867 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.436845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.438060 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.438043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.445155 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.445132    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncrh\" (UniqueName: \"kubernetes.io/projected/70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a-kube-api-access-jncrh\") pod \"alertmanager-main-0\" (UID: \"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.506322 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.506300    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff56b308-d378-4bd2-9db2-4264b8053e41" path="/var/lib/kubelet/pods/ff56b308-d378-4bd2-9db2-4264b8053e41/volumes"
Sep 17 17:36:48.560578 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.560560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:48.839090 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:48.839062    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:48.842285 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:36:48.842259    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ff6b12_8cc9_4d90_a4d5_8f8d99522a4a.slice/crio-ab7eff6f33b4ba6e568cf7666157877d115261981af180eeb363d68a946757c3 WatchSource:0}: Error finding container ab7eff6f33b4ba6e568cf7666157877d115261981af180eeb363d68a946757c3: Status 404 returned error can't find the container with id ab7eff6f33b4ba6e568cf7666157877d115261981af180eeb363d68a946757c3
Sep 17 17:36:49.196488 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:49.196397    2566 generic.go:356] "Generic (PLEG): container finished" podID="70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a" containerID="9bbfe843b531fca38c7eab0c8708c055182dc419b5c0e6b3296e325b1e595d16" exitCode=0
Sep 17 17:36:49.196639 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:49.196478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerDied","Data":"9bbfe843b531fca38c7eab0c8708c055182dc419b5c0e6b3296e325b1e595d16"}
Sep 17 17:36:49.196639 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:49.196590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerStarted","Data":"ab7eff6f33b4ba6e568cf7666157877d115261981af180eeb363d68a946757c3"}
Sep 17 17:36:50.204010 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.203979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerStarted","Data":"150f7734f7bf87716385ea32515548ef842a3704ca5338259ef1029dd6a40282"}
Sep 17 17:36:50.204368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.204040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerStarted","Data":"c10fa9256dd8b73c1606a9d28824a82ce0bc34af6e38000b053fe791b741381b"}
Sep 17 17:36:50.204368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.204070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerStarted","Data":"2cc8141ae35416216dd1bb1a8fb0e83dd77024610df7e8549592e94635c9ae7b"}
Sep 17 17:36:50.204368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.204093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerStarted","Data":"9608c51514c3e2919ad43e3de8c77dab796af8e3df1961e0f8cfe5be0e807311"}
Sep 17 17:36:50.204368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.204114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerStarted","Data":"476ce306635ae207a5f09489962ba59fa16603e3f76bd5f0cefebb4c9112b9c2"}
Sep 17 17:36:50.204368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.204138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a","Type":"ContainerStarted","Data":"68147a4a968e3089afc2988632e65dca303b601c25ef5a6ec0eaaf16f63fd3e6"}
Sep 17 17:36:50.234300 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.234250    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.219778815 podStartE2EDuration="2.23423367s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.454855ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:50.232418237 +0000 UTC m=+170.092009594" watchObservedRunningTime="2026-09-17 17:36:50.23423367 +0000 UTC m=+170.093825025"
Sep 17 17:36:50.553562 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.553533    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c87696bbc-wz6sm"]
Sep 17 17:36:50.556731 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.556715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.559753 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.559734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:36:50.559881 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.559759    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:36:50.559881 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.559773    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:36:50.559881 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.559784    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:36:50.559881 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.559846    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-726st\""
Sep 17 17:36:50.560259 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.560240    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:36:50.564639 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.564621    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:36:50.575053 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.575033    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c87696bbc-wz6sm"]
Sep 17 17:36:50.649649 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-metrics-client-ca\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.649756 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-federate-client-tls\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.649756 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-secret-telemeter-client\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.649756 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-serving-certs-ca-bundle\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.649756 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.649880 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd494\" (UniqueName: \"kubernetes.io/projected/1c2dda93-3811-41ed-8b56-ac82582ffdeb-kube-api-access-jd494\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.649880 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.649880 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.649850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-telemeter-client-tls\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750758 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-metrics-client-ca\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-federate-client-tls\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-secret-telemeter-client\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-serving-certs-ca-bundle\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750982 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750982 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd494\" (UniqueName: \"kubernetes.io/projected/1c2dda93-3811-41ed-8b56-ac82582ffdeb-kube-api-access-jd494\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750982 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.750982 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.750932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-telemeter-client-tls\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.751614 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.751585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-serving-certs-ca-bundle\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.751703 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.751587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-metrics-client-ca\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.752051 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.752030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2dda93-3811-41ed-8b56-ac82582ffdeb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.753365 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.753346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-federate-client-tls\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.753590 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.753570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.753876 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.753857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-secret-telemeter-client\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.753930 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.753857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2dda93-3811-41ed-8b56-ac82582ffdeb-telemeter-client-tls\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.760484 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.760464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd494\" (UniqueName: \"kubernetes.io/projected/1c2dda93-3811-41ed-8b56-ac82582ffdeb-kube-api-access-jd494\") pod \"telemeter-client-c87696bbc-wz6sm\" (UID: \"1c2dda93-3811-41ed-8b56-ac82582ffdeb\") " pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.867105 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.867045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm"
Sep 17 17:36:50.887980 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.887956    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:50.888370 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.888315    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="prometheus" containerID="cri-o://cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d" gracePeriod=600
Sep 17 17:36:50.888447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.888359    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8" gracePeriod=600
Sep 17 17:36:50.888447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.888387    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="thanos-sidecar" containerID="cri-o://6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9" gracePeriod=600
Sep 17 17:36:50.888447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.888387    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-web" containerID="cri-o://e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27" gracePeriod=600
Sep 17 17:36:50.888606 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.888447    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="config-reloader" containerID="cri-o://c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795" gracePeriod=600
Sep 17 17:36:50.888606 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:50.888489    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy" containerID="cri-o://5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74" gracePeriod=600
Sep 17 17:36:51.127889 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.127864    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.147261 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.147209    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c87696bbc-wz6sm"]
Sep 17 17:36:51.149160 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:36:51.149134    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2dda93_3811_41ed_8b56_ac82582ffdeb.slice/crio-7a873d2992433752b8dd0d08d57f8b610cb53111d61610d7c410dc61d454c048 WatchSource:0}: Error finding container 7a873d2992433752b8dd0d08d57f8b610cb53111d61610d7c410dc61d454c048: Status 404 returned error can't find the container with id 7a873d2992433752b8dd0d08d57f8b610cb53111d61610d7c410dc61d454c048
Sep 17 17:36:51.211634 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.211600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm" event={"ID":"1c2dda93-3811-41ed-8b56-ac82582ffdeb","Type":"ContainerStarted","Data":"7a873d2992433752b8dd0d08d57f8b610cb53111d61610d7c410dc61d454c048"}
Sep 17 17:36:51.215237 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215201    2566 generic.go:356] "Generic (PLEG): container finished" podID="514016aa-8741-41fb-b156-cd480ae00172" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8" exitCode=0
Sep 17 17:36:51.215237 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215227    2566 generic.go:356] "Generic (PLEG): container finished" podID="514016aa-8741-41fb-b156-cd480ae00172" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74" exitCode=0
Sep 17 17:36:51.215237 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215237    2566 generic.go:356] "Generic (PLEG): container finished" podID="514016aa-8741-41fb-b156-cd480ae00172" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27" exitCode=0
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215246    2566 generic.go:356] "Generic (PLEG): container finished" podID="514016aa-8741-41fb-b156-cd480ae00172" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9" exitCode=0
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215254    2566 generic.go:356] "Generic (PLEG): container finished" podID="514016aa-8741-41fb-b156-cd480ae00172" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795" exitCode=0
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215264    2566 generic.go:356] "Generic (PLEG): container finished" podID="514016aa-8741-41fb-b156-cd480ae00172" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d" exitCode=0
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"}
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215303    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"}
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"}
Sep 17 17:36:51.215422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"}
Sep 17 17:36:51.215693 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"}
Sep 17 17:36:51.215693 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"}
Sep 17 17:36:51.215693 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215416    2566 scope.go:117] "RemoveContainer" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.215693 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.215492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"514016aa-8741-41fb-b156-cd480ae00172","Type":"ContainerDied","Data":"7ddb0768da450ec97917da45231c309de3e6f262f63752764cf67c5bae542be5"}
Sep 17 17:36:51.223181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.223167    2566 scope.go:117] "RemoveContainer" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.229812 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.229798    2566 scope.go:117] "RemoveContainer" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.235861 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.235846    2566 scope.go:117] "RemoveContainer" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.241889 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.241877    2566 scope.go:117] "RemoveContainer" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.247918 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.247891    2566 scope.go:117] "RemoveContainer" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.254487 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.254469    2566 scope.go:117] "RemoveContainer" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.256505 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-trusted-ca-bundle\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256527    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-2\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256607 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-thanos-prometheus-http-client-file\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256654 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-1\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256654 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256672    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-metrics-client-ca\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256703    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-kubelet-serving-ca-bundle\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-tls\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-kube-rbac-proxy\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256836    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-serving-certs-ca-bundle\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256862    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-db\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.256945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256878    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:51.256945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-config\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:51.257180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256961    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:51.257180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.256972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-tls-assets\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257032    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-config-out\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257068    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-web-config\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-metrics-client-certs\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257180 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-grpc-tls\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257484 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257218    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-0\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257484 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257484 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257281    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crnm9\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-kube-api-access-crnm9\") pod \"514016aa-8741-41fb-b156-cd480ae00172\" (UID: \"514016aa-8741-41fb-b156-cd480ae00172\") "
Sep 17 17:36:51.257484 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257388    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:51.257741 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257562    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.257741 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257583    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.257741 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257599    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.257741 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257615    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.258010 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.257980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:51.258750 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.258725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:51.259089 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.259063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.259777 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.259388    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:51.259777 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.259521    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:51.260512 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.260436    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.260512 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.260457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:51.260648 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.260551    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.261097 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.261071    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-kube-api-access-crnm9" (OuterVolumeSpecName: "kube-api-access-crnm9") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "kube-api-access-crnm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:51.261647 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.261312    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.261647 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.261376    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-config" (OuterVolumeSpecName: "config") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.261647 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.261600    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.261879 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.261793    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.262470 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.262444    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-config-out" (OuterVolumeSpecName: "config-out") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:51.262547 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.262477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.264127 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.264111    2566 scope.go:117] "RemoveContainer" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.264384 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:51.264365    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": container with ID starting with bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8 not found: ID does not exist" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.264429 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.264392    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"} err="failed to get container status \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": rpc error: code = NotFound desc = could not find container \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": container with ID starting with bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8 not found: ID does not exist"
Sep 17 17:36:51.264429 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.264408    2566 scope.go:117] "RemoveContainer" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.264665 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:51.264646    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": container with ID starting with 5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74 not found: ID does not exist" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.264713 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.264669    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"} err="failed to get container status \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": rpc error: code = NotFound desc = could not find container \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": container with ID starting with 5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74 not found: ID does not exist"
Sep 17 17:36:51.264713 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.264685    2566 scope.go:117] "RemoveContainer" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.264943 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:51.264904    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": container with ID starting with e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27 not found: ID does not exist" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.265045 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.264945    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"} err="failed to get container status \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": rpc error: code = NotFound desc = could not find container \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": container with ID starting with e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27 not found: ID does not exist"
Sep 17 17:36:51.265045 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.264964    2566 scope.go:117] "RemoveContainer" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.265164 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:51.265147    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": container with ID starting with 6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9 not found: ID does not exist" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.265222 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265160    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"} err="failed to get container status \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": rpc error: code = NotFound desc = could not find container \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": container with ID starting with 6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9 not found: ID does not exist"
Sep 17 17:36:51.265222 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265171    2566 scope.go:117] "RemoveContainer" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.265386 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:51.265365    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": container with ID starting with c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795 not found: ID does not exist" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.265448 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265392    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"} err="failed to get container status \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": rpc error: code = NotFound desc = could not find container \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": container with ID starting with c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795 not found: ID does not exist"
Sep 17 17:36:51.265448 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265407    2566 scope.go:117] "RemoveContainer" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.265664 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:51.265646    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": container with ID starting with cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d not found: ID does not exist" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.265708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265667    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"} err="failed to get container status \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": rpc error: code = NotFound desc = could not find container \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": container with ID starting with cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d not found: ID does not exist"
Sep 17 17:36:51.265708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265680    2566 scope.go:117] "RemoveContainer" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.265954 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:36:51.265934    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": container with ID starting with c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23 not found: ID does not exist" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.266020 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265956    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"} err="failed to get container status \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": rpc error: code = NotFound desc = could not find container \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": container with ID starting with c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23 not found: ID does not exist"
Sep 17 17:36:51.266020 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.265975    2566 scope.go:117] "RemoveContainer" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.266204 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266189    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"} err="failed to get container status \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": rpc error: code = NotFound desc = could not find container \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": container with ID starting with bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8 not found: ID does not exist"
Sep 17 17:36:51.266245 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266204    2566 scope.go:117] "RemoveContainer" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.266402 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266383    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"} err="failed to get container status \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": rpc error: code = NotFound desc = could not find container \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": container with ID starting with 5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74 not found: ID does not exist"
Sep 17 17:36:51.266468 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266403    2566 scope.go:117] "RemoveContainer" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.266618 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266603    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"} err="failed to get container status \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": rpc error: code = NotFound desc = could not find container \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": container with ID starting with e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27 not found: ID does not exist"
Sep 17 17:36:51.266682 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266619    2566 scope.go:117] "RemoveContainer" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.266867 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266849    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"} err="failed to get container status \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": rpc error: code = NotFound desc = could not find container \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": container with ID starting with 6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9 not found: ID does not exist"
Sep 17 17:36:51.266964 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.266868    2566 scope.go:117] "RemoveContainer" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.267140 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267123    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"} err="failed to get container status \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": rpc error: code = NotFound desc = could not find container \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": container with ID starting with c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795 not found: ID does not exist"
Sep 17 17:36:51.267181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267141    2566 scope.go:117] "RemoveContainer" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.267413 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267396    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"} err="failed to get container status \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": rpc error: code = NotFound desc = could not find container \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": container with ID starting with cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d not found: ID does not exist"
Sep 17 17:36:51.267474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267414    2566 scope.go:117] "RemoveContainer" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.267649 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267634    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"} err="failed to get container status \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": rpc error: code = NotFound desc = could not find container \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": container with ID starting with c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23 not found: ID does not exist"
Sep 17 17:36:51.267705 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267650    2566 scope.go:117] "RemoveContainer" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.267884 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267867    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"} err="failed to get container status \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": rpc error: code = NotFound desc = could not find container \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": container with ID starting with bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8 not found: ID does not exist"
Sep 17 17:36:51.267971 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.267884    2566 scope.go:117] "RemoveContainer" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.268139 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268122    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"} err="failed to get container status \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": rpc error: code = NotFound desc = could not find container \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": container with ID starting with 5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74 not found: ID does not exist"
Sep 17 17:36:51.268184 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268140    2566 scope.go:117] "RemoveContainer" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.268364 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268346    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"} err="failed to get container status \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": rpc error: code = NotFound desc = could not find container \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": container with ID starting with e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27 not found: ID does not exist"
Sep 17 17:36:51.268417 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268364    2566 scope.go:117] "RemoveContainer" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.268586 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268571    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"} err="failed to get container status \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": rpc error: code = NotFound desc = could not find container \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": container with ID starting with 6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9 not found: ID does not exist"
Sep 17 17:36:51.268641 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268585    2566 scope.go:117] "RemoveContainer" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.268785 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268770    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"} err="failed to get container status \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": rpc error: code = NotFound desc = could not find container \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": container with ID starting with c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795 not found: ID does not exist"
Sep 17 17:36:51.268785 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268784    2566 scope.go:117] "RemoveContainer" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.269011 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.268996    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"} err="failed to get container status \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": rpc error: code = NotFound desc = could not find container \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": container with ID starting with cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d not found: ID does not exist"
Sep 17 17:36:51.269011 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269010    2566 scope.go:117] "RemoveContainer" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.269223 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269208    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"} err="failed to get container status \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": rpc error: code = NotFound desc = could not find container \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": container with ID starting with c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23 not found: ID does not exist"
Sep 17 17:36:51.269293 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269223    2566 scope.go:117] "RemoveContainer" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.269462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269443    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"} err="failed to get container status \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": rpc error: code = NotFound desc = could not find container \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": container with ID starting with bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8 not found: ID does not exist"
Sep 17 17:36:51.269518 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269463    2566 scope.go:117] "RemoveContainer" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.269704 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269690    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"} err="failed to get container status \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": rpc error: code = NotFound desc = could not find container \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": container with ID starting with 5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74 not found: ID does not exist"
Sep 17 17:36:51.269746 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269704    2566 scope.go:117] "RemoveContainer" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.269933 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269897    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"} err="failed to get container status \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": rpc error: code = NotFound desc = could not find container \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": container with ID starting with e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27 not found: ID does not exist"
Sep 17 17:36:51.269993 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.269933    2566 scope.go:117] "RemoveContainer" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.270133 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270116    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"} err="failed to get container status \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": rpc error: code = NotFound desc = could not find container \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": container with ID starting with 6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9 not found: ID does not exist"
Sep 17 17:36:51.270174 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270132    2566 scope.go:117] "RemoveContainer" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.270350 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270332    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"} err="failed to get container status \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": rpc error: code = NotFound desc = could not find container \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": container with ID starting with c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795 not found: ID does not exist"
Sep 17 17:36:51.270386 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270351    2566 scope.go:117] "RemoveContainer" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.270569 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270552    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"} err="failed to get container status \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": rpc error: code = NotFound desc = could not find container \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": container with ID starting with cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d not found: ID does not exist"
Sep 17 17:36:51.270569 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270568    2566 scope.go:117] "RemoveContainer" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.270785 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270768    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"} err="failed to get container status \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": rpc error: code = NotFound desc = could not find container \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": container with ID starting with c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23 not found: ID does not exist"
Sep 17 17:36:51.270821 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.270786    2566 scope.go:117] "RemoveContainer" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.271041 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271025    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"} err="failed to get container status \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": rpc error: code = NotFound desc = could not find container \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": container with ID starting with bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8 not found: ID does not exist"
Sep 17 17:36:51.271084 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271042    2566 scope.go:117] "RemoveContainer" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.271275 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271258    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"} err="failed to get container status \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": rpc error: code = NotFound desc = could not find container \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": container with ID starting with 5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74 not found: ID does not exist"
Sep 17 17:36:51.271325 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271275    2566 scope.go:117] "RemoveContainer" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.271498 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271482    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"} err="failed to get container status \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": rpc error: code = NotFound desc = could not find container \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": container with ID starting with e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27 not found: ID does not exist"
Sep 17 17:36:51.271535 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271499    2566 scope.go:117] "RemoveContainer" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.271711 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271696    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"} err="failed to get container status \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": rpc error: code = NotFound desc = could not find container \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": container with ID starting with 6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9 not found: ID does not exist"
Sep 17 17:36:51.271763 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271711    2566 scope.go:117] "RemoveContainer" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.271902 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271887    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"} err="failed to get container status \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": rpc error: code = NotFound desc = could not find container \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": container with ID starting with c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795 not found: ID does not exist"
Sep 17 17:36:51.271955 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.271902    2566 scope.go:117] "RemoveContainer" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.272181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272161    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"} err="failed to get container status \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": rpc error: code = NotFound desc = could not find container \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": container with ID starting with cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d not found: ID does not exist"
Sep 17 17:36:51.272181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272180    2566 scope.go:117] "RemoveContainer" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.272424 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272405    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"} err="failed to get container status \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": rpc error: code = NotFound desc = could not find container \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": container with ID starting with c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23 not found: ID does not exist"
Sep 17 17:36:51.272424 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272422    2566 scope.go:117] "RemoveContainer" containerID="bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"
Sep 17 17:36:51.272655 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272640    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8"} err="failed to get container status \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": rpc error: code = NotFound desc = could not find container \"bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8\": container with ID starting with bafcbfe1b8c309cc1b3a11544c0b327c0065b84009eaac1e3529f1ea3565bde8 not found: ID does not exist"
Sep 17 17:36:51.272655 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272655    2566 scope.go:117] "RemoveContainer" containerID="5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"
Sep 17 17:36:51.272879 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272861    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74"} err="failed to get container status \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": rpc error: code = NotFound desc = could not find container \"5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74\": container with ID starting with 5a84cc4d61aab313ad5256eb7f0fe1568dc8b45c72cb42c7ba6d7ffde7996d74 not found: ID does not exist"
Sep 17 17:36:51.272978 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.272880    2566 scope.go:117] "RemoveContainer" containerID="e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"
Sep 17 17:36:51.273184 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273159    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27"} err="failed to get container status \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": rpc error: code = NotFound desc = could not find container \"e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27\": container with ID starting with e052fd19b4bc0e369493b4a7a8a74daef3233d3ed77145f984403d04f245ef27 not found: ID does not exist"
Sep 17 17:36:51.273234 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273184    2566 scope.go:117] "RemoveContainer" containerID="6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"
Sep 17 17:36:51.273409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273390    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9"} err="failed to get container status \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": rpc error: code = NotFound desc = could not find container \"6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9\": container with ID starting with 6f8bb569e4a35985958c41306aa98dfce4370940b91091bf88e22dc45f1e16e9 not found: ID does not exist"
Sep 17 17:36:51.273409 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273409    2566 scope.go:117] "RemoveContainer" containerID="c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"
Sep 17 17:36:51.273650 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273633    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795"} err="failed to get container status \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": rpc error: code = NotFound desc = could not find container \"c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795\": container with ID starting with c544e78ad53bef9d8dcab08f6c3e393c4f07e8dacca86a1ec366e0976e4c5795 not found: ID does not exist"
Sep 17 17:36:51.273650 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273651    2566 scope.go:117] "RemoveContainer" containerID="cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"
Sep 17 17:36:51.273870 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273855    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d"} err="failed to get container status \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": rpc error: code = NotFound desc = could not find container \"cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d\": container with ID starting with cbf99d085018fd65b744c2813a8870885b05145d1d442881294700b7f6d6fb1d not found: ID does not exist"
Sep 17 17:36:51.273957 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.273871    2566 scope.go:117] "RemoveContainer" containerID="c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"
Sep 17 17:36:51.274079 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.274060    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-web-config" (OuterVolumeSpecName: "web-config") pod "514016aa-8741-41fb-b156-cd480ae00172" (UID: "514016aa-8741-41fb-b156-cd480ae00172"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:51.274125 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.274112    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23"} err="failed to get container status \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": rpc error: code = NotFound desc = could not find container \"c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23\": container with ID starting with c37f3d74383e926c4077d2926b2c41e3f927d61239587621bd5b5fe4af392d23 not found: ID does not exist"
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359013    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-kube-rbac-proxy\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359043    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-db\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359058    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-config\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359073    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-tls-assets\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359087    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/514016aa-8741-41fb-b156-cd480ae00172-config-out\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359101    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-web-config\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359115    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-metrics-client-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359128    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-grpc-tls\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359162    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359185    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359204    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crnm9\" (UniqueName: \"kubernetes.io/projected/514016aa-8741-41fb-b156-cd480ae00172-kube-api-access-crnm9\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359221    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359236    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359253    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-metrics-client-ca\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359267    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514016aa-8741-41fb-b156-cd480ae00172-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.359298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.359283    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/514016aa-8741-41fb-b156-cd480ae00172-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:36:51.544285 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.544257    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:51.551874 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.551847    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.582948    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583314    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="config-reloader"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583330    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="config-reloader"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583344    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583351    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583363    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="prometheus"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583372    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="prometheus"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583387    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-web"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583395    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-web"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583405    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="thanos-sidecar"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583413    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="thanos-sidecar"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583427    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="init-config-reloader"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583435    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="init-config-reloader"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583445    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-thanos"
Sep 17 17:36:51.583474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583452    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-thanos"
Sep 17 17:36:51.584170 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583527    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="prometheus"
Sep 17 17:36:51.584170 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583540    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-web"
Sep 17 17:36:51.584170 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583549    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy-thanos"
Sep 17 17:36:51.584170 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583558    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="thanos-sidecar"
Sep 17 17:36:51.584170 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583568    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="config-reloader"
Sep 17 17:36:51.584170 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.583576    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="514016aa-8741-41fb-b156-cd480ae00172" containerName="kube-rbac-proxy"
Sep 17 17:36:51.604312 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.604293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.607446 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.607426    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:36:51.607658 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.607642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:36:51.607734 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.607665    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:36:51.608698 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.608639    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ll73hqjiajmk\""
Sep 17 17:36:51.608840 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.608719    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:36:51.608840 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.608747    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:36:51.609009 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.608853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:36:51.609009 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.608878    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:36:51.609206 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609168    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:36:51.609417 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609222    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:36:51.609417 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:36:51.609417 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609311    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:36:51.609597 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609417    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:51.609597 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609467    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:36:51.609597 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609467    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:36:51.609875 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.609849    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zscrq\""
Sep 17 17:36:51.611290 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.611271    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:36:51.613301 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.613282    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:36:51.761991 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.761960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762107 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.761996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762107 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762107 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762107 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-config\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762289 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762289 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762289 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762289 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rcq\" (UniqueName: \"kubernetes.io/projected/00878956-76d9-480f-bb14-543a8c5bc19a-kube-api-access-h5rcq\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762289 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00878956-76d9-480f-bb14-543a8c5bc19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762807 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762807 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00878956-76d9-480f-bb14-543a8c5bc19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762807 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.762807 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.762596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.863752 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00878956-76d9-480f-bb14-543a8c5bc19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.863866 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.863866 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.863866 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.863866 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00878956-76d9-480f-bb14-543a8c5bc19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.863866 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864080 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864080 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864080 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864080 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.863975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864080 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864080 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-config\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rcq\" (UniqueName: \"kubernetes.io/projected/00878956-76d9-480f-bb14-543a8c5bc19a-kube-api-access-h5rcq\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864351 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.864754 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.864593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.866560 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.865003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.866560 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.865756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.866560 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.866244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867029 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.866999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867135 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867191 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00878956-76d9-480f-bb14-543a8c5bc19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867224    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00878956-76d9-480f-bb14-543a8c5bc19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867407 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867984 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867611    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867984 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.867984 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.867898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.868555 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.868534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.868804 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.868788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-config\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.869734 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.869714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.869875 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.869852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.869997 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.869982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00878956-76d9-480f-bb14-543a8c5bc19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.870702 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.870683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/00878956-76d9-480f-bb14-543a8c5bc19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.876001 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.875982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rcq\" (UniqueName: \"kubernetes.io/projected/00878956-76d9-480f-bb14-543a8c5bc19a-kube-api-access-h5rcq\") pod \"prometheus-k8s-0\" (UID: \"00878956-76d9-480f-bb14-543a8c5bc19a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:51.916081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:51.916059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:52.200996 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:52.200967    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:52.202827 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:36:52.202798    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00878956_76d9_480f_bb14_543a8c5bc19a.slice/crio-741bf65fee7cea86ebe96f6121b1c0ae1f6d91a1031ef24b8417836c450aa900 WatchSource:0}: Error finding container 741bf65fee7cea86ebe96f6121b1c0ae1f6d91a1031ef24b8417836c450aa900: Status 404 returned error can't find the container with id 741bf65fee7cea86ebe96f6121b1c0ae1f6d91a1031ef24b8417836c450aa900
Sep 17 17:36:52.220396 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:52.220368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerStarted","Data":"741bf65fee7cea86ebe96f6121b1c0ae1f6d91a1031ef24b8417836c450aa900"}
Sep 17 17:36:52.505874 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:52.505804    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="514016aa-8741-41fb-b156-cd480ae00172" path="/var/lib/kubelet/pods/514016aa-8741-41fb-b156-cd480ae00172/volumes"
Sep 17 17:36:53.230250 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:53.230213    2566 generic.go:356] "Generic (PLEG): container finished" podID="00878956-76d9-480f-bb14-543a8c5bc19a" containerID="dd63b78edf0e14c34ce69ce9f3baab874ad8ff8793ad13875a03325ed4e6eed2" exitCode=0
Sep 17 17:36:53.230694 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:53.230313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerDied","Data":"dd63b78edf0e14c34ce69ce9f3baab874ad8ff8793ad13875a03325ed4e6eed2"}
Sep 17 17:36:54.236974 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.236923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm" event={"ID":"1c2dda93-3811-41ed-8b56-ac82582ffdeb","Type":"ContainerStarted","Data":"f6d564937c06762f9ecd7ecda23b8cd23d3bf58acf1e9a2ff371afc9d64f5bba"}
Sep 17 17:36:54.237348 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.236999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm" event={"ID":"1c2dda93-3811-41ed-8b56-ac82582ffdeb","Type":"ContainerStarted","Data":"1070b1dbe3e7bc03406636e5613a9be60b2fd1434f196a3b4e110d76e22c6a94"}
Sep 17 17:36:54.237348 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.237019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm" event={"ID":"1c2dda93-3811-41ed-8b56-ac82582ffdeb","Type":"ContainerStarted","Data":"bf3f0cb1bfc20cd793ad013449c3d1c7b67d2295a92d0eb72f8ec493fe596781"}
Sep 17 17:36:54.240164 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.240136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerStarted","Data":"0bae82e6c50b8ff369acbd0a20f1319fc4ee33214c2f35ddf9f0ce687a102fb0"}
Sep 17 17:36:54.240261 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.240189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerStarted","Data":"fdb280c04bf6d7941c699a05582d0e55ee32d554602fbf2c1ed96ee3278944cd"}
Sep 17 17:36:54.240261 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.240219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerStarted","Data":"d03cea3c1d806b66ba5f7431881735b942689bc5bac229488ecbe9aee77d91f4"}
Sep 17 17:36:54.240261 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.240249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerStarted","Data":"93a9d98fc55ac14bd93a804c27f4f4f37929a3756231f975ab5189103cf380eb"}
Sep 17 17:36:54.240363 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.240273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerStarted","Data":"ef9bb6c5d681b3188c8e20ab38b9361180d31e06429755ca4752abe8c9aca1b5"}
Sep 17 17:36:54.240363 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.240297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"00878956-76d9-480f-bb14-543a8c5bc19a","Type":"ContainerStarted","Data":"3bfd761583ead33185e67642348f7b4c2f4639186f01516c2e9a5e0aa0c35faf"}
Sep 17 17:36:54.262637 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.262595    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c87696bbc-wz6sm" podStartSLOduration=1.996893537 podStartE2EDuration="4.262581775s" totalImagesPullingTime="2.265688238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:54.261928703 +0000 UTC m=+174.121520050" watchObservedRunningTime="2026-09-17 17:36:54.262581775 +0000 UTC m=+174.122173131"
Sep 17 17:36:54.297557 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:54.297515    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.277618361 podStartE2EDuration="3.297502416s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.884055ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:54.296149707 +0000 UTC m=+174.155741099" watchObservedRunningTime="2026-09-17 17:36:54.297502416 +0000 UTC m=+174.157093772"
Sep 17 17:36:56.916250 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:36:56.916211    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:51.917298 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:37:51.917268    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:51.932539 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:37:51.932510    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:52.462575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:37:52.462551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:02.367119 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:39:02.367093    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:39:02.370715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:39:02.370696    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:39:02.372517 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:39:02.372499    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:43:09.801079 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.800995    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c42f7"]
Sep 17 17:43:09.804028 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.804012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:09.806610 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.806587    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 17:43:09.807537 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.807520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8zdsm\""
Sep 17 17:43:09.807637 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.807558    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 17:43:09.812275 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.812250    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c42f7"]
Sep 17 17:43:09.904893 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.904846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947c6c1e-5f42-4be0-8f98-26abe852fc63-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c42f7\" (UID: \"947c6c1e-5f42-4be0-8f98-26abe852fc63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:09.905086 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:09.904905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5sdb\" (UniqueName: \"kubernetes.io/projected/947c6c1e-5f42-4be0-8f98-26abe852fc63-kube-api-access-c5sdb\") pod \"cert-manager-webhook-6486ffd997-c42f7\" (UID: \"947c6c1e-5f42-4be0-8f98-26abe852fc63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:10.006314 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.006274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947c6c1e-5f42-4be0-8f98-26abe852fc63-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c42f7\" (UID: \"947c6c1e-5f42-4be0-8f98-26abe852fc63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:10.006555 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.006334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5sdb\" (UniqueName: \"kubernetes.io/projected/947c6c1e-5f42-4be0-8f98-26abe852fc63-kube-api-access-c5sdb\") pod \"cert-manager-webhook-6486ffd997-c42f7\" (UID: \"947c6c1e-5f42-4be0-8f98-26abe852fc63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:10.016653 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.016624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947c6c1e-5f42-4be0-8f98-26abe852fc63-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c42f7\" (UID: \"947c6c1e-5f42-4be0-8f98-26abe852fc63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:10.019535 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.019506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5sdb\" (UniqueName: \"kubernetes.io/projected/947c6c1e-5f42-4be0-8f98-26abe852fc63-kube-api-access-c5sdb\") pod \"cert-manager-webhook-6486ffd997-c42f7\" (UID: \"947c6c1e-5f42-4be0-8f98-26abe852fc63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:10.122447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.122353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:10.392178 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.392154    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c42f7"]
Sep 17 17:43:10.394616 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:43:10.394590    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod947c6c1e_5f42_4be0_8f98_26abe852fc63.slice/crio-1b729daa9a5fb4f35b361c780dfb67d6c059839a8452b42ded75d48b3f887e54 WatchSource:0}: Error finding container 1b729daa9a5fb4f35b361c780dfb67d6c059839a8452b42ded75d48b3f887e54: Status 404 returned error can't find the container with id 1b729daa9a5fb4f35b361c780dfb67d6c059839a8452b42ded75d48b3f887e54
Sep 17 17:43:10.396489 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.396473    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:43:10.616583 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:10.616547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7" event={"ID":"947c6c1e-5f42-4be0-8f98-26abe852fc63","Type":"ContainerStarted","Data":"1b729daa9a5fb4f35b361c780dfb67d6c059839a8452b42ded75d48b3f887e54"}
Sep 17 17:43:14.633078 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:14.633043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7" event={"ID":"947c6c1e-5f42-4be0-8f98-26abe852fc63","Type":"ContainerStarted","Data":"c2ac4914954bd1ed3a8ca286b8cf6c8b5ec1540e77d0a0da1a25725e859d9136"}
Sep 17 17:43:14.633453 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:14.633121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:14.651529 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:14.651487    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7" podStartSLOduration=2.421771561 podStartE2EDuration="5.651462693s" totalImagesPullingTime="3.229691132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:14.650280533 +0000 UTC m=+554.509871889" watchObservedRunningTime="2026-09-17 17:43:14.651462693 +0000 UTC m=+554.511054049"
Sep 17 17:43:20.639475 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:20.639442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c42f7"
Sep 17 17:43:27.981462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:27.981427    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dhjvx"]
Sep 17 17:43:27.987755 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:27.987733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:27.990398 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:27.990376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-btm78\""
Sep 17 17:43:27.994617 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:27.994277    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dhjvx"]
Sep 17 17:43:28.061849 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.061807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hnz\" (UniqueName: \"kubernetes.io/projected/545b45fb-5210-4311-87d9-6716e7ac9d63-kube-api-access-69hnz\") pod \"cert-manager-55b4d954d5-dhjvx\" (UID: \"545b45fb-5210-4311-87d9-6716e7ac9d63\") " pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:28.062052 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.061926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/545b45fb-5210-4311-87d9-6716e7ac9d63-bound-sa-token\") pod \"cert-manager-55b4d954d5-dhjvx\" (UID: \"545b45fb-5210-4311-87d9-6716e7ac9d63\") " pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:28.162596 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.162548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/545b45fb-5210-4311-87d9-6716e7ac9d63-bound-sa-token\") pod \"cert-manager-55b4d954d5-dhjvx\" (UID: \"545b45fb-5210-4311-87d9-6716e7ac9d63\") " pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:28.162709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.162622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hnz\" (UniqueName: \"kubernetes.io/projected/545b45fb-5210-4311-87d9-6716e7ac9d63-kube-api-access-69hnz\") pod \"cert-manager-55b4d954d5-dhjvx\" (UID: \"545b45fb-5210-4311-87d9-6716e7ac9d63\") " pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:28.172633 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.172597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/545b45fb-5210-4311-87d9-6716e7ac9d63-bound-sa-token\") pod \"cert-manager-55b4d954d5-dhjvx\" (UID: \"545b45fb-5210-4311-87d9-6716e7ac9d63\") " pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:28.172743 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.172724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hnz\" (UniqueName: \"kubernetes.io/projected/545b45fb-5210-4311-87d9-6716e7ac9d63-kube-api-access-69hnz\") pod \"cert-manager-55b4d954d5-dhjvx\" (UID: \"545b45fb-5210-4311-87d9-6716e7ac9d63\") " pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:28.297788 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.297676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dhjvx"
Sep 17 17:43:28.568638 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.568606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dhjvx"]
Sep 17 17:43:28.571925 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:43:28.571887    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545b45fb_5210_4311_87d9_6716e7ac9d63.slice/crio-830d0bd4a797c11c73737a664e01b5aec43054ca9434264d625b039dbcbb798b WatchSource:0}: Error finding container 830d0bd4a797c11c73737a664e01b5aec43054ca9434264d625b039dbcbb798b: Status 404 returned error can't find the container with id 830d0bd4a797c11c73737a664e01b5aec43054ca9434264d625b039dbcbb798b
Sep 17 17:43:28.686305 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.686266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dhjvx" event={"ID":"545b45fb-5210-4311-87d9-6716e7ac9d63","Type":"ContainerStarted","Data":"31934189524f544b77b800830ea69d9e426a778d82c6d344ff0b556b7429539d"}
Sep 17 17:43:28.686305 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.686305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dhjvx" event={"ID":"545b45fb-5210-4311-87d9-6716e7ac9d63","Type":"ContainerStarted","Data":"830d0bd4a797c11c73737a664e01b5aec43054ca9434264d625b039dbcbb798b"}
Sep 17 17:43:28.703446 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:43:28.703395    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dhjvx" podStartSLOduration=1.703376502 podStartE2EDuration="1.703376502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:28.702516785 +0000 UTC m=+568.562108144" watchObservedRunningTime="2026-09-17 17:43:28.703376502 +0000 UTC m=+568.562967857"
Sep 17 17:44:02.400205 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:02.400180    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:44:02.401591 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:02.401573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:44:06.091381 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.091346    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"]
Sep 17 17:44:06.093877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.093861    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.096720 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.096696    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 17:44:06.097882 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.097863    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 17:44:06.098115 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.098101    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 17:44:06.098199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.098182    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:44:06.098246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.098197    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6n2s5\""
Sep 17 17:44:06.098246 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.098235    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 17:44:06.104308 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.104285    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"]
Sep 17 17:44:06.178773 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.178733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da095f24-21d6-48c0-bebf-73425f088141-cert\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.178983 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.178791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da095f24-21d6-48c0-bebf-73425f088141-metrics-cert\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.178983 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.178852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nchfb\" (UniqueName: \"kubernetes.io/projected/da095f24-21d6-48c0-bebf-73425f088141-kube-api-access-nchfb\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.179071 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.178982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da095f24-21d6-48c0-bebf-73425f088141-manager-config\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.280153 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.280113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da095f24-21d6-48c0-bebf-73425f088141-manager-config\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.280287 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.280164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da095f24-21d6-48c0-bebf-73425f088141-cert\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.280287 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.280194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da095f24-21d6-48c0-bebf-73425f088141-metrics-cert\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.280287 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.280219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nchfb\" (UniqueName: \"kubernetes.io/projected/da095f24-21d6-48c0-bebf-73425f088141-kube-api-access-nchfb\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.280771 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.280744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da095f24-21d6-48c0-bebf-73425f088141-manager-config\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.282726 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.282701    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da095f24-21d6-48c0-bebf-73425f088141-cert\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.282815 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.282760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da095f24-21d6-48c0-bebf-73425f088141-metrics-cert\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.289758 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.289736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nchfb\" (UniqueName: \"kubernetes.io/projected/da095f24-21d6-48c0-bebf-73425f088141-kube-api-access-nchfb\") pod \"lws-controller-manager-ff6d497d7-54kmf\" (UID: \"da095f24-21d6-48c0-bebf-73425f088141\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.402519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.402421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:06.675729 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.675692    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"]
Sep 17 17:44:06.677801 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:44:06.677777    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda095f24_21d6_48c0_bebf_73425f088141.slice/crio-ec78b9ee8301ccadc754c7c8bb9b4cf7da169e2a24fe1e956b92a492193550c7 WatchSource:0}: Error finding container ec78b9ee8301ccadc754c7c8bb9b4cf7da169e2a24fe1e956b92a492193550c7: Status 404 returned error can't find the container with id ec78b9ee8301ccadc754c7c8bb9b4cf7da169e2a24fe1e956b92a492193550c7
Sep 17 17:44:06.834639 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:06.834605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf" event={"ID":"da095f24-21d6-48c0-bebf-73425f088141","Type":"ContainerStarted","Data":"ec78b9ee8301ccadc754c7c8bb9b4cf7da169e2a24fe1e956b92a492193550c7"}
Sep 17 17:44:09.850236 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:09.850200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf" event={"ID":"da095f24-21d6-48c0-bebf-73425f088141","Type":"ContainerStarted","Data":"1f3c857eeaf578c89742c14859ec5f5fc4d274f80b0174e0d52101a32635075f"}
Sep 17 17:44:09.850637 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:09.850263    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:44:09.875387 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:09.875337    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf" podStartSLOduration=1.437955101 podStartE2EDuration="3.875321006s" totalImagesPullingTime="2.437365905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:09.873417373 +0000 UTC m=+609.733008741" watchObservedRunningTime="2026-09-17 17:44:09.875321006 +0000 UTC m=+609.734912363"
Sep 17 17:44:20.856511 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:44:20.856431    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-54kmf"
Sep 17 17:45:10.797401 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.797364    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qcl7s"]
Sep 17 17:45:10.800527 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.800511    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:10.802955 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.802933    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wrlsg\""
Sep 17 17:45:10.803043 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.802945    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 17:45:10.803868 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.803852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 17:45:10.803930 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.803867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 17:45:10.808881 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.808858    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qcl7s"]
Sep 17 17:45:10.897033 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.897005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qcl7s"]
Sep 17 17:45:10.925811 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.925782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5124de6f-b941-44a8-b67d-cbe013aa09df-config-file\") pod \"limitador-limitador-8588749977-qcl7s\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") " pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:10.925980 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:10.925833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sz8\" (UniqueName: \"kubernetes.io/projected/5124de6f-b941-44a8-b67d-cbe013aa09df-kube-api-access-45sz8\") pod \"limitador-limitador-8588749977-qcl7s\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") " pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:11.026588 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.026563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sz8\" (UniqueName: \"kubernetes.io/projected/5124de6f-b941-44a8-b67d-cbe013aa09df-kube-api-access-45sz8\") pod \"limitador-limitador-8588749977-qcl7s\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") " pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:11.026737 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.026622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5124de6f-b941-44a8-b67d-cbe013aa09df-config-file\") pod \"limitador-limitador-8588749977-qcl7s\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") " pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:11.027198 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.027182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5124de6f-b941-44a8-b67d-cbe013aa09df-config-file\") pod \"limitador-limitador-8588749977-qcl7s\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") " pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:11.038810 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.038779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sz8\" (UniqueName: \"kubernetes.io/projected/5124de6f-b941-44a8-b67d-cbe013aa09df-kube-api-access-45sz8\") pod \"limitador-limitador-8588749977-qcl7s\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") " pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:11.111254 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.111229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:11.387094 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.387070    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qcl7s"]
Sep 17 17:45:11.389617 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:45:11.389579    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5124de6f_b941_44a8_b67d_cbe013aa09df.slice/crio-cffa23261177dfd1cced71d1f5207a40634e0687762f64a3f9b5ff394131f242 WatchSource:0}: Error finding container cffa23261177dfd1cced71d1f5207a40634e0687762f64a3f9b5ff394131f242: Status 404 returned error can't find the container with id cffa23261177dfd1cced71d1f5207a40634e0687762f64a3f9b5ff394131f242
Sep 17 17:45:11.853131 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.853105    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-4s9ng"]
Sep 17 17:45:11.857557 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.857540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4s9ng"
Sep 17 17:45:11.859893 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.859866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9htfg\""
Sep 17 17:45:11.861142 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.861122    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4s9ng"]
Sep 17 17:45:11.933340 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:11.933314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7n4\" (UniqueName: \"kubernetes.io/projected/d1d26f09-7c38-4ce5-b2fb-0ca05158df54-kube-api-access-9n7n4\") pod \"authorino-59c974bd48-4s9ng\" (UID: \"d1d26f09-7c38-4ce5-b2fb-0ca05158df54\") " pod="kuadrant-system/authorino-59c974bd48-4s9ng"
Sep 17 17:45:12.034378 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:12.034350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7n4\" (UniqueName: \"kubernetes.io/projected/d1d26f09-7c38-4ce5-b2fb-0ca05158df54-kube-api-access-9n7n4\") pod \"authorino-59c974bd48-4s9ng\" (UID: \"d1d26f09-7c38-4ce5-b2fb-0ca05158df54\") " pod="kuadrant-system/authorino-59c974bd48-4s9ng"
Sep 17 17:45:12.043373 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:12.043347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7n4\" (UniqueName: \"kubernetes.io/projected/d1d26f09-7c38-4ce5-b2fb-0ca05158df54-kube-api-access-9n7n4\") pod \"authorino-59c974bd48-4s9ng\" (UID: \"d1d26f09-7c38-4ce5-b2fb-0ca05158df54\") " pod="kuadrant-system/authorino-59c974bd48-4s9ng"
Sep 17 17:45:12.101598 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:12.101570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s" event={"ID":"5124de6f-b941-44a8-b67d-cbe013aa09df","Type":"ContainerStarted","Data":"cffa23261177dfd1cced71d1f5207a40634e0687762f64a3f9b5ff394131f242"}
Sep 17 17:45:12.167858 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:12.167803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4s9ng"
Sep 17 17:45:12.442645 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:12.442622    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4s9ng"]
Sep 17 17:45:12.445189 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:45:12.445154    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1d26f09_7c38_4ce5_b2fb_0ca05158df54.slice/crio-fb668b20b97d67832cc6abb753be225e32e611df28ccef584e3b7fae3ec98171 WatchSource:0}: Error finding container fb668b20b97d67832cc6abb753be225e32e611df28ccef584e3b7fae3ec98171: Status 404 returned error can't find the container with id fb668b20b97d67832cc6abb753be225e32e611df28ccef584e3b7fae3ec98171
Sep 17 17:45:13.107445 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:13.107414    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4s9ng" event={"ID":"d1d26f09-7c38-4ce5-b2fb-0ca05158df54","Type":"ContainerStarted","Data":"fb668b20b97d67832cc6abb753be225e32e611df28ccef584e3b7fae3ec98171"}
Sep 17 17:45:14.113159 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:14.113120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s" event={"ID":"5124de6f-b941-44a8-b67d-cbe013aa09df","Type":"ContainerStarted","Data":"1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad"}
Sep 17 17:45:14.113586 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:14.113264    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:14.132345 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:14.132294    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s" podStartSLOduration=1.7804385169999999 podStartE2EDuration="4.132275053s" totalImagesPullingTime="2.351836536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:14.13084196 +0000 UTC m=+673.990433317" watchObservedRunningTime="2026-09-17 17:45:14.132275053 +0000 UTC m=+673.991866411"
Sep 17 17:45:15.118242 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:15.118203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4s9ng" event={"ID":"d1d26f09-7c38-4ce5-b2fb-0ca05158df54","Type":"ContainerStarted","Data":"7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4"}
Sep 17 17:45:15.134776 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:15.134723    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-4s9ng" podStartSLOduration=1.885786132 podStartE2EDuration="4.134707615s" totalImagesPullingTime="2.248921483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:15.133395612 +0000 UTC m=+674.992986968" watchObservedRunningTime="2026-09-17 17:45:15.134707615 +0000 UTC m=+674.994298971"
Sep 17 17:45:25.119570 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:25.119536    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:27.457235 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:27.457184    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qcl7s"]
Sep 17 17:45:27.457676 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:27.457446    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s" podUID="5124de6f-b941-44a8-b67d-cbe013aa09df" containerName="limitador" containerID="cri-o://1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad" gracePeriod=30
Sep 17 17:45:28.004401 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.004379    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:28.170727 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.170698    2566 generic.go:356] "Generic (PLEG): container finished" podID="5124de6f-b941-44a8-b67d-cbe013aa09df" containerID="1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad" exitCode=0
Sep 17 17:45:28.170944 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.170782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s" event={"ID":"5124de6f-b941-44a8-b67d-cbe013aa09df","Type":"ContainerDied","Data":"1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad"}
Sep 17 17:45:28.170944 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.170799    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s"
Sep 17 17:45:28.170944 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.170820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qcl7s" event={"ID":"5124de6f-b941-44a8-b67d-cbe013aa09df","Type":"ContainerDied","Data":"cffa23261177dfd1cced71d1f5207a40634e0687762f64a3f9b5ff394131f242"}
Sep 17 17:45:28.170944 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.170848    2566 scope.go:117] "RemoveContainer" containerID="1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad"
Sep 17 17:45:28.174524 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.174497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45sz8\" (UniqueName: \"kubernetes.io/projected/5124de6f-b941-44a8-b67d-cbe013aa09df-kube-api-access-45sz8\") pod \"5124de6f-b941-44a8-b67d-cbe013aa09df\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") "
Sep 17 17:45:28.174602 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.174558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5124de6f-b941-44a8-b67d-cbe013aa09df-config-file\") pod \"5124de6f-b941-44a8-b67d-cbe013aa09df\" (UID: \"5124de6f-b941-44a8-b67d-cbe013aa09df\") "
Sep 17 17:45:28.174887 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.174862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5124de6f-b941-44a8-b67d-cbe013aa09df-config-file" (OuterVolumeSpecName: "config-file") pod "5124de6f-b941-44a8-b67d-cbe013aa09df" (UID: "5124de6f-b941-44a8-b67d-cbe013aa09df"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:45:28.176616 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.176592    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5124de6f-b941-44a8-b67d-cbe013aa09df-kube-api-access-45sz8" (OuterVolumeSpecName: "kube-api-access-45sz8") pod "5124de6f-b941-44a8-b67d-cbe013aa09df" (UID: "5124de6f-b941-44a8-b67d-cbe013aa09df"). InnerVolumeSpecName "kube-api-access-45sz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:28.179058 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.179038    2566 scope.go:117] "RemoveContainer" containerID="1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad"
Sep 17 17:45:28.179328 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:45:28.179309    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad\": container with ID starting with 1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad not found: ID does not exist" containerID="1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad"
Sep 17 17:45:28.179394 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.179335    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad"} err="failed to get container status \"1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad\": rpc error: code = NotFound desc = could not find container \"1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad\": container with ID starting with 1a425277c2ef729a7ca6871be36f1b6dd01dd42990a8bf3eb1b8beb199ca20ad not found: ID does not exist"
Sep 17 17:45:28.275376 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.275338    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5124de6f-b941-44a8-b67d-cbe013aa09df-config-file\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:45:28.275376 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.275371    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45sz8\" (UniqueName: \"kubernetes.io/projected/5124de6f-b941-44a8-b67d-cbe013aa09df-kube-api-access-45sz8\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:45:28.505869 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.505842    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qcl7s"]
Sep 17 17:45:28.508755 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:28.508727    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qcl7s"]
Sep 17 17:45:30.506627 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:30.506591    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5124de6f-b941-44a8-b67d-cbe013aa09df" path="/var/lib/kubelet/pods/5124de6f-b941-44a8-b67d-cbe013aa09df/volumes"
Sep 17 17:45:37.562446 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:37.562410    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4s9ng"]
Sep 17 17:45:37.562868 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:37.562577    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-4s9ng" podUID="d1d26f09-7c38-4ce5-b2fb-0ca05158df54" containerName="authorino" containerID="cri-o://7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4" gracePeriod=30
Sep 17 17:45:37.813562 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:37.813507    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4s9ng"
Sep 17 17:45:37.861290 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:37.861264    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n7n4\" (UniqueName: \"kubernetes.io/projected/d1d26f09-7c38-4ce5-b2fb-0ca05158df54-kube-api-access-9n7n4\") pod \"d1d26f09-7c38-4ce5-b2fb-0ca05158df54\" (UID: \"d1d26f09-7c38-4ce5-b2fb-0ca05158df54\") "
Sep 17 17:45:37.863241 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:37.863210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d26f09-7c38-4ce5-b2fb-0ca05158df54-kube-api-access-9n7n4" (OuterVolumeSpecName: "kube-api-access-9n7n4") pod "d1d26f09-7c38-4ce5-b2fb-0ca05158df54" (UID: "d1d26f09-7c38-4ce5-b2fb-0ca05158df54"). InnerVolumeSpecName "kube-api-access-9n7n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:37.962404 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:37.962365    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n7n4\" (UniqueName: \"kubernetes.io/projected/d1d26f09-7c38-4ce5-b2fb-0ca05158df54-kube-api-access-9n7n4\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:45:38.210181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.210091    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1d26f09-7c38-4ce5-b2fb-0ca05158df54" containerID="7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4" exitCode=0
Sep 17 17:45:38.210181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.210134    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4s9ng"
Sep 17 17:45:38.210355 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.210178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4s9ng" event={"ID":"d1d26f09-7c38-4ce5-b2fb-0ca05158df54","Type":"ContainerDied","Data":"7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4"}
Sep 17 17:45:38.210355 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.210218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4s9ng" event={"ID":"d1d26f09-7c38-4ce5-b2fb-0ca05158df54","Type":"ContainerDied","Data":"fb668b20b97d67832cc6abb753be225e32e611df28ccef584e3b7fae3ec98171"}
Sep 17 17:45:38.210355 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.210235    2566 scope.go:117] "RemoveContainer" containerID="7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4"
Sep 17 17:45:38.218274 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.218252    2566 scope.go:117] "RemoveContainer" containerID="7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4"
Sep 17 17:45:38.218507 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:45:38.218492    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4\": container with ID starting with 7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4 not found: ID does not exist" containerID="7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4"
Sep 17 17:45:38.218554 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.218514    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4"} err="failed to get container status \"7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4\": rpc error: code = NotFound desc = could not find container \"7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4\": container with ID starting with 7e4e713d057721a888c9f534906c84bcb55f64d70eeead67248411b07a43cee4 not found: ID does not exist"
Sep 17 17:45:38.232242 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.232218    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4s9ng"]
Sep 17 17:45:38.239240 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.239221    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4s9ng"]
Sep 17 17:45:38.505629 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:45:38.505550    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d26f09-7c38-4ce5-b2fb-0ca05158df54" path="/var/lib/kubelet/pods/d1d26f09-7c38-4ce5-b2fb-0ca05158df54/volumes"
Sep 17 17:46:06.266782 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.266749    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-9bw7h"]
Sep 17 17:46:06.267181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.267100    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5124de6f-b941-44a8-b67d-cbe013aa09df" containerName="limitador"
Sep 17 17:46:06.267181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.267113    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5124de6f-b941-44a8-b67d-cbe013aa09df" containerName="limitador"
Sep 17 17:46:06.267181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.267129    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d26f09-7c38-4ce5-b2fb-0ca05158df54" containerName="authorino"
Sep 17 17:46:06.267181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.267135    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d26f09-7c38-4ce5-b2fb-0ca05158df54" containerName="authorino"
Sep 17 17:46:06.267307 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.267184    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d26f09-7c38-4ce5-b2fb-0ca05158df54" containerName="authorino"
Sep 17 17:46:06.267307 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.267194    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5124de6f-b941-44a8-b67d-cbe013aa09df" containerName="limitador"
Sep 17 17:46:06.273718 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.273695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.276348 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.276320    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:46:06.276487 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.276321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:46:06.277507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.277343    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-j7gck\""
Sep 17 17:46:06.277507 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.277347    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:46:06.278081 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.278059    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-9bw7h"]
Sep 17 17:46:06.281457 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.281110    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-77658fbd58-w4mjv"]
Sep 17 17:46:06.284605 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.284585    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:06.286925 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.286890    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 17:46:06.287036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.286960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-x2nm6\""
Sep 17 17:46:06.294938 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.294903    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-77658fbd58-w4mjv"]
Sep 17 17:46:06.305466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.305445    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-wbpfh"]
Sep 17 17:46:06.308558 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.308543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.310922 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.310896    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z6254\""
Sep 17 17:46:06.310922 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.310905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:46:06.319641 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.319620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wbpfh"]
Sep 17 17:46:06.396169 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.396140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpdl\" (UniqueName: \"kubernetes.io/projected/75c9dd75-d94d-4263-aeaf-44d2579cb943-kube-api-access-ccpdl\") pod \"seaweedfs-64568bcd49-wbpfh\" (UID: \"75c9dd75-d94d-4263-aeaf-44d2579cb943\") " pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.396309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.396178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56lj\" (UniqueName: \"kubernetes.io/projected/572aec9b-9e51-4595-ad22-f5de1d8862aa-kube-api-access-w56lj\") pod \"llmisvc-controller-manager-77658fbd58-w4mjv\" (UID: \"572aec9b-9e51-4595-ad22-f5de1d8862aa\") " pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:06.396309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.396209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ad8681-031c-43f2-aa6f-9960936d76a4-cert\") pod \"kserve-controller-manager-7f654675b-9bw7h\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") " pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.396309 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.396263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/572aec9b-9e51-4595-ad22-f5de1d8862aa-cert\") pod \"llmisvc-controller-manager-77658fbd58-w4mjv\" (UID: \"572aec9b-9e51-4595-ad22-f5de1d8862aa\") " pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:06.396446 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.396339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75c9dd75-d94d-4263-aeaf-44d2579cb943-data\") pod \"seaweedfs-64568bcd49-wbpfh\" (UID: \"75c9dd75-d94d-4263-aeaf-44d2579cb943\") " pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.396446 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.396362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp942\" (UniqueName: \"kubernetes.io/projected/68ad8681-031c-43f2-aa6f-9960936d76a4-kube-api-access-xp942\") pod \"kserve-controller-manager-7f654675b-9bw7h\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") " pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.497135 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.497113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpdl\" (UniqueName: \"kubernetes.io/projected/75c9dd75-d94d-4263-aeaf-44d2579cb943-kube-api-access-ccpdl\") pod \"seaweedfs-64568bcd49-wbpfh\" (UID: \"75c9dd75-d94d-4263-aeaf-44d2579cb943\") " pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.497256 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.497146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56lj\" (UniqueName: \"kubernetes.io/projected/572aec9b-9e51-4595-ad22-f5de1d8862aa-kube-api-access-w56lj\") pod \"llmisvc-controller-manager-77658fbd58-w4mjv\" (UID: \"572aec9b-9e51-4595-ad22-f5de1d8862aa\") " pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:06.497256 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.497166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ad8681-031c-43f2-aa6f-9960936d76a4-cert\") pod \"kserve-controller-manager-7f654675b-9bw7h\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") " pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.497256 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.497179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/572aec9b-9e51-4595-ad22-f5de1d8862aa-cert\") pod \"llmisvc-controller-manager-77658fbd58-w4mjv\" (UID: \"572aec9b-9e51-4595-ad22-f5de1d8862aa\") " pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:06.497256 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.497241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75c9dd75-d94d-4263-aeaf-44d2579cb943-data\") pod \"seaweedfs-64568bcd49-wbpfh\" (UID: \"75c9dd75-d94d-4263-aeaf-44d2579cb943\") " pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.497461 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:46:06.497268    2566 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 17:46:06.497461 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.497285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp942\" (UniqueName: \"kubernetes.io/projected/68ad8681-031c-43f2-aa6f-9960936d76a4-kube-api-access-xp942\") pod \"kserve-controller-manager-7f654675b-9bw7h\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") " pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.497461 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:46:06.497331    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572aec9b-9e51-4595-ad22-f5de1d8862aa-cert podName:572aec9b-9e51-4595-ad22-f5de1d8862aa nodeName:}" failed. No retries permitted until 2026-09-17 17:46:06.997311301 +0000 UTC m=+726.856902636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/572aec9b-9e51-4595-ad22-f5de1d8862aa-cert") pod "llmisvc-controller-manager-77658fbd58-w4mjv" (UID: "572aec9b-9e51-4595-ad22-f5de1d8862aa") : secret "llmisvc-webhook-server-cert" not found
Sep 17 17:46:06.497626 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.497579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75c9dd75-d94d-4263-aeaf-44d2579cb943-data\") pod \"seaweedfs-64568bcd49-wbpfh\" (UID: \"75c9dd75-d94d-4263-aeaf-44d2579cb943\") " pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.499653 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.499635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ad8681-031c-43f2-aa6f-9960936d76a4-cert\") pod \"kserve-controller-manager-7f654675b-9bw7h\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") " pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.508404 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.508378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp942\" (UniqueName: \"kubernetes.io/projected/68ad8681-031c-43f2-aa6f-9960936d76a4-kube-api-access-xp942\") pod \"kserve-controller-manager-7f654675b-9bw7h\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") " pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.509168 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.509147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56lj\" (UniqueName: \"kubernetes.io/projected/572aec9b-9e51-4595-ad22-f5de1d8862aa-kube-api-access-w56lj\") pod \"llmisvc-controller-manager-77658fbd58-w4mjv\" (UID: \"572aec9b-9e51-4595-ad22-f5de1d8862aa\") " pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:06.509867 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.509847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpdl\" (UniqueName: \"kubernetes.io/projected/75c9dd75-d94d-4263-aeaf-44d2579cb943-kube-api-access-ccpdl\") pod \"seaweedfs-64568bcd49-wbpfh\" (UID: \"75c9dd75-d94d-4263-aeaf-44d2579cb943\") " pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.584775 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.584754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:06.617760 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.617736    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:06.858779 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.858756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-9bw7h"]
Sep 17 17:46:06.860948 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:46:06.860921    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ad8681_031c_43f2_aa6f_9960936d76a4.slice/crio-3a9c3cef25e6a7ee228b06d0a5bdd88d1ce980af6df6a765de18b798f05387a4 WatchSource:0}: Error finding container 3a9c3cef25e6a7ee228b06d0a5bdd88d1ce980af6df6a765de18b798f05387a4: Status 404 returned error can't find the container with id 3a9c3cef25e6a7ee228b06d0a5bdd88d1ce980af6df6a765de18b798f05387a4
Sep 17 17:46:06.897758 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:06.897737    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wbpfh"]
Sep 17 17:46:06.899259 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:46:06.899234    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c9dd75_d94d_4263_aeaf_44d2579cb943.slice/crio-fffd8f706f2a312f3ca49c48914904cea64ee49859696fe6699895c430ddd280 WatchSource:0}: Error finding container fffd8f706f2a312f3ca49c48914904cea64ee49859696fe6699895c430ddd280: Status 404 returned error can't find the container with id fffd8f706f2a312f3ca49c48914904cea64ee49859696fe6699895c430ddd280
Sep 17 17:46:07.001860 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:07.001827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/572aec9b-9e51-4595-ad22-f5de1d8862aa-cert\") pod \"llmisvc-controller-manager-77658fbd58-w4mjv\" (UID: \"572aec9b-9e51-4595-ad22-f5de1d8862aa\") " pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:07.004103 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:07.004081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/572aec9b-9e51-4595-ad22-f5de1d8862aa-cert\") pod \"llmisvc-controller-manager-77658fbd58-w4mjv\" (UID: \"572aec9b-9e51-4595-ad22-f5de1d8862aa\") " pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:07.194145 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:07.194073    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:07.326248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:07.326215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-9bw7h" event={"ID":"68ad8681-031c-43f2-aa6f-9960936d76a4","Type":"ContainerStarted","Data":"3a9c3cef25e6a7ee228b06d0a5bdd88d1ce980af6df6a765de18b798f05387a4"}
Sep 17 17:46:07.327336 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:07.327307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wbpfh" event={"ID":"75c9dd75-d94d-4263-aeaf-44d2579cb943","Type":"ContainerStarted","Data":"fffd8f706f2a312f3ca49c48914904cea64ee49859696fe6699895c430ddd280"}
Sep 17 17:46:07.467793 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:07.467750    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-77658fbd58-w4mjv"]
Sep 17 17:46:07.470330 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:46:07.470289    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod572aec9b_9e51_4595_ad22_f5de1d8862aa.slice/crio-cbf0a45f7898ddbb962b366707caacd83eec80dd4dac511fd86027ad431f2993 WatchSource:0}: Error finding container cbf0a45f7898ddbb962b366707caacd83eec80dd4dac511fd86027ad431f2993: Status 404 returned error can't find the container with id cbf0a45f7898ddbb962b366707caacd83eec80dd4dac511fd86027ad431f2993
Sep 17 17:46:08.341446 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:08.341407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv" event={"ID":"572aec9b-9e51-4595-ad22-f5de1d8862aa","Type":"ContainerStarted","Data":"cbf0a45f7898ddbb962b366707caacd83eec80dd4dac511fd86027ad431f2993"}
Sep 17 17:46:12.361413 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.361375    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wbpfh" event={"ID":"75c9dd75-d94d-4263-aeaf-44d2579cb943","Type":"ContainerStarted","Data":"96cca93d1c26e0051d04a1001c6d63931b01212b46b40ea364da6491ffd0113b"}
Sep 17 17:46:12.361783 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.361439    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:12.362805 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.362778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-9bw7h" event={"ID":"68ad8681-031c-43f2-aa6f-9960936d76a4","Type":"ContainerStarted","Data":"13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a"}
Sep 17 17:46:12.362900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.362840    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:12.364249 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.364229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv" event={"ID":"572aec9b-9e51-4595-ad22-f5de1d8862aa","Type":"ContainerStarted","Data":"83766beb216d1a5d2ab11c93f49c56253310098a720291fb09856e13b5560b9f"}
Sep 17 17:46:12.364355 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.364342    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:12.382226 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.382184    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-wbpfh" podStartSLOduration=1.883494213 podStartE2EDuration="6.382169914s" totalImagesPullingTime="4.498675701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:12.379940687 +0000 UTC m=+732.239532040" watchObservedRunningTime="2026-09-17 17:46:12.382169914 +0000 UTC m=+732.241761263"
Sep 17 17:46:12.401573 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.401527    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv" podStartSLOduration=2.522539142 podStartE2EDuration="6.401512616s" totalImagesPullingTime="3.878973474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:12.400185779 +0000 UTC m=+732.259777136" watchObservedRunningTime="2026-09-17 17:46:12.401512616 +0000 UTC m=+732.261103991"
Sep 17 17:46:12.422681 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:12.422640    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-9bw7h" podStartSLOduration=2.012085212 podStartE2EDuration="6.422628288s" totalImagesPullingTime="4.410543076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:12.420981229 +0000 UTC m=+732.280572586" watchObservedRunningTime="2026-09-17 17:46:12.422628288 +0000 UTC m=+732.282219643"
Sep 17 17:46:18.370780 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:18.370746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-wbpfh"
Sep 17 17:46:43.370604 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:43.370572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-77658fbd58-w4mjv"
Sep 17 17:46:43.373628 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:43.373609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:50.844866 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:50.844828    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-9bw7h"]
Sep 17 17:46:50.845342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:50.845086    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7f654675b-9bw7h" podUID="68ad8681-031c-43f2-aa6f-9960936d76a4" containerName="manager" containerID="cri-o://13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a" gracePeriod=10
Sep 17 17:46:50.870169 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:50.870136    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-5qgdq"]
Sep 17 17:46:50.872765 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:50.872743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:50.884062 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:50.884026    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-5qgdq"]
Sep 17 17:46:50.975533 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:50.975495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae7f6847-d7ff-4976-824a-f55af7f07f22-cert\") pod \"kserve-controller-manager-7f654675b-5qgdq\" (UID: \"ae7f6847-d7ff-4976-824a-f55af7f07f22\") " pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:50.975689 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:50.975631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtq77\" (UniqueName: \"kubernetes.io/projected/ae7f6847-d7ff-4976-824a-f55af7f07f22-kube-api-access-xtq77\") pod \"kserve-controller-manager-7f654675b-5qgdq\" (UID: \"ae7f6847-d7ff-4976-824a-f55af7f07f22\") " pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:51.076758 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.076726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtq77\" (UniqueName: \"kubernetes.io/projected/ae7f6847-d7ff-4976-824a-f55af7f07f22-kube-api-access-xtq77\") pod \"kserve-controller-manager-7f654675b-5qgdq\" (UID: \"ae7f6847-d7ff-4976-824a-f55af7f07f22\") " pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:51.076952 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.076777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae7f6847-d7ff-4976-824a-f55af7f07f22-cert\") pod \"kserve-controller-manager-7f654675b-5qgdq\" (UID: \"ae7f6847-d7ff-4976-824a-f55af7f07f22\") " pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:51.079389 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.079367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae7f6847-d7ff-4976-824a-f55af7f07f22-cert\") pod \"kserve-controller-manager-7f654675b-5qgdq\" (UID: \"ae7f6847-d7ff-4976-824a-f55af7f07f22\") " pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:51.079757 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.079734    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:51.085873 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.085848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtq77\" (UniqueName: \"kubernetes.io/projected/ae7f6847-d7ff-4976-824a-f55af7f07f22-kube-api-access-xtq77\") pod \"kserve-controller-manager-7f654675b-5qgdq\" (UID: \"ae7f6847-d7ff-4976-824a-f55af7f07f22\") " pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:51.178159 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.178006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ad8681-031c-43f2-aa6f-9960936d76a4-cert\") pod \"68ad8681-031c-43f2-aa6f-9960936d76a4\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") "
Sep 17 17:46:51.178311 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.178179    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp942\" (UniqueName: \"kubernetes.io/projected/68ad8681-031c-43f2-aa6f-9960936d76a4-kube-api-access-xp942\") pod \"68ad8681-031c-43f2-aa6f-9960936d76a4\" (UID: \"68ad8681-031c-43f2-aa6f-9960936d76a4\") "
Sep 17 17:46:51.180199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.180166    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ad8681-031c-43f2-aa6f-9960936d76a4-cert" (OuterVolumeSpecName: "cert") pod "68ad8681-031c-43f2-aa6f-9960936d76a4" (UID: "68ad8681-031c-43f2-aa6f-9960936d76a4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:51.180311 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.180221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ad8681-031c-43f2-aa6f-9960936d76a4-kube-api-access-xp942" (OuterVolumeSpecName: "kube-api-access-xp942") pod "68ad8681-031c-43f2-aa6f-9960936d76a4" (UID: "68ad8681-031c-43f2-aa6f-9960936d76a4"). InnerVolumeSpecName "kube-api-access-xp942". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:51.220594 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.220561    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:51.279554 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.279508    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp942\" (UniqueName: \"kubernetes.io/projected/68ad8681-031c-43f2-aa6f-9960936d76a4-kube-api-access-xp942\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:46:51.279554 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.279535    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ad8681-031c-43f2-aa6f-9960936d76a4-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:46:51.491124 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.491100    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-5qgdq"]
Sep 17 17:46:51.493193 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:46:51.493167    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7f6847_d7ff_4976_824a_f55af7f07f22.slice/crio-e1b6f43a8766c931a56ca7dfad9a758cf4baffc7d998b84e1d371161fd7f2f6f WatchSource:0}: Error finding container e1b6f43a8766c931a56ca7dfad9a758cf4baffc7d998b84e1d371161fd7f2f6f: Status 404 returned error can't find the container with id e1b6f43a8766c931a56ca7dfad9a758cf4baffc7d998b84e1d371161fd7f2f6f
Sep 17 17:46:51.521131 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.521090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-5qgdq" event={"ID":"ae7f6847-d7ff-4976-824a-f55af7f07f22","Type":"ContainerStarted","Data":"e1b6f43a8766c931a56ca7dfad9a758cf4baffc7d998b84e1d371161fd7f2f6f"}
Sep 17 17:46:51.522328 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.522305    2566 generic.go:356] "Generic (PLEG): container finished" podID="68ad8681-031c-43f2-aa6f-9960936d76a4" containerID="13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a" exitCode=0
Sep 17 17:46:51.522467 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.522343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-9bw7h" event={"ID":"68ad8681-031c-43f2-aa6f-9960936d76a4","Type":"ContainerDied","Data":"13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a"}
Sep 17 17:46:51.522467 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.522358    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-9bw7h"
Sep 17 17:46:51.522467 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.522373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-9bw7h" event={"ID":"68ad8681-031c-43f2-aa6f-9960936d76a4","Type":"ContainerDied","Data":"3a9c3cef25e6a7ee228b06d0a5bdd88d1ce980af6df6a765de18b798f05387a4"}
Sep 17 17:46:51.522467 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.522397    2566 scope.go:117] "RemoveContainer" containerID="13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a"
Sep 17 17:46:51.531800 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.531779    2566 scope.go:117] "RemoveContainer" containerID="13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a"
Sep 17 17:46:51.532201 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:46:51.532180    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a\": container with ID starting with 13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a not found: ID does not exist" containerID="13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a"
Sep 17 17:46:51.532283 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.532214    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a"} err="failed to get container status \"13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a\": rpc error: code = NotFound desc = could not find container \"13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a\": container with ID starting with 13d53d59f0e44d57abe9d0f2353325561dd294d70a79e46b71dac18bd855d72a not found: ID does not exist"
Sep 17 17:46:51.547195 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.547160    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-9bw7h"]
Sep 17 17:46:51.553477 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:51.553448    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-9bw7h"]
Sep 17 17:46:52.506606 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:52.506572    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ad8681-031c-43f2-aa6f-9960936d76a4" path="/var/lib/kubelet/pods/68ad8681-031c-43f2-aa6f-9960936d76a4/volumes"
Sep 17 17:46:52.528362 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:52.528332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-5qgdq" event={"ID":"ae7f6847-d7ff-4976-824a-f55af7f07f22","Type":"ContainerStarted","Data":"ab31cc97a0ab938afe4ce6f861aa016f718b81c9b18f489ae96b56fd7725c76d"}
Sep 17 17:46:52.528525 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:52.528464    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:46:52.547821 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:46:52.547766    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-5qgdq" podStartSLOduration=2.293922315 podStartE2EDuration="2.547749008s" totalImagesPullingTime="253.826693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:52.546201201 +0000 UTC m=+772.405792570" watchObservedRunningTime="2026-09-17 17:46:52.547749008 +0000 UTC m=+772.407340364"
Sep 17 17:47:23.536645 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:23.536615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-5qgdq"
Sep 17 17:47:42.963583 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:42.963547    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-wpj5d"]
Sep 17 17:47:42.963983 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:42.963893    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ad8681-031c-43f2-aa6f-9960936d76a4" containerName="manager"
Sep 17 17:47:42.963983 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:42.963921    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ad8681-031c-43f2-aa6f-9960936d76a4" containerName="manager"
Sep 17 17:47:42.964058 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:42.963989    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ad8681-031c-43f2-aa6f-9960936d76a4" containerName="manager"
Sep 17 17:47:42.965973 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:42.965957    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wpj5d"
Sep 17 17:47:42.968272 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:42.968253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:47:42.973974 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:42.973947    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-wpj5d"]
Sep 17 17:47:43.016255 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.016216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6vqh\" (UniqueName: \"kubernetes.io/projected/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-kube-api-access-m6vqh\") pod \"s3-init-wpj5d\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") " pod="kserve/s3-init-wpj5d"
Sep 17 17:47:43.016422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.016318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-model-cache\") pod \"s3-init-wpj5d\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") " pod="kserve/s3-init-wpj5d"
Sep 17 17:47:43.117553 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.117510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-model-cache\") pod \"s3-init-wpj5d\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") " pod="kserve/s3-init-wpj5d"
Sep 17 17:47:43.117704 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.117584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6vqh\" (UniqueName: \"kubernetes.io/projected/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-kube-api-access-m6vqh\") pod \"s3-init-wpj5d\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") " pod="kserve/s3-init-wpj5d"
Sep 17 17:47:43.117965 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.117945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-model-cache\") pod \"s3-init-wpj5d\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") " pod="kserve/s3-init-wpj5d"
Sep 17 17:47:43.126751 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.126717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6vqh\" (UniqueName: \"kubernetes.io/projected/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-kube-api-access-m6vqh\") pod \"s3-init-wpj5d\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") " pod="kserve/s3-init-wpj5d"
Sep 17 17:47:43.276268 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.276178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wpj5d"
Sep 17 17:47:43.546558 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.546534    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-wpj5d"]
Sep 17 17:47:43.548371 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:47:43.548350    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6efe2f4c_63e7_4e75_9bdb_170767d4e22c.slice/crio-2fae070b2267b2681c5d9cc0479714927f4b627da4f6cf989042b29c44fb87a8 WatchSource:0}: Error finding container 2fae070b2267b2681c5d9cc0479714927f4b627da4f6cf989042b29c44fb87a8: Status 404 returned error can't find the container with id 2fae070b2267b2681c5d9cc0479714927f4b627da4f6cf989042b29c44fb87a8
Sep 17 17:47:43.735265 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:43.735229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wpj5d" event={"ID":"6efe2f4c-63e7-4e75-9bdb-170767d4e22c","Type":"ContainerStarted","Data":"2fae070b2267b2681c5d9cc0479714927f4b627da4f6cf989042b29c44fb87a8"}
Sep 17 17:47:47.757996 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:47.757960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wpj5d" event={"ID":"6efe2f4c-63e7-4e75-9bdb-170767d4e22c","Type":"ContainerStarted","Data":"029fa49abe29f9af0480461aaa11bb4b7cfba6d5bc8a0a70c2f6c8bf05156c53"}
Sep 17 17:47:49.769272 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:49.769240    2566 generic.go:356] "Generic (PLEG): container finished" podID="6efe2f4c-63e7-4e75-9bdb-170767d4e22c" containerID="029fa49abe29f9af0480461aaa11bb4b7cfba6d5bc8a0a70c2f6c8bf05156c53" exitCode=0
Sep 17 17:47:49.769644 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:49.769320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wpj5d" event={"ID":"6efe2f4c-63e7-4e75-9bdb-170767d4e22c","Type":"ContainerDied","Data":"029fa49abe29f9af0480461aaa11bb4b7cfba6d5bc8a0a70c2f6c8bf05156c53"}
Sep 17 17:47:54.796499 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:54.796462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wpj5d" event={"ID":"6efe2f4c-63e7-4e75-9bdb-170767d4e22c","Type":"ContainerStarted","Data":"e6d98c1fe9ee20339bf7d0803e7e8df0cfa0615cad821309ea9bd9db2ab20779"}
Sep 17 17:47:54.814576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:47:54.814523    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-wpj5d" podStartSLOduration=2.990275898 podStartE2EDuration="12.814506803s" totalImagesPullingTime="7.998681976s" totalInitContainerRuntime="1.825548929s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:54.812732615 +0000 UTC m=+834.672323971" watchObservedRunningTime="2026-09-17 17:47:54.814506803 +0000 UTC m=+834.674098158"
Sep 17 17:49:00.070884 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:00.070844    2566 generic.go:356] "Generic (PLEG): container finished" podID="6efe2f4c-63e7-4e75-9bdb-170767d4e22c" containerID="e6d98c1fe9ee20339bf7d0803e7e8df0cfa0615cad821309ea9bd9db2ab20779" exitCode=0
Sep 17 17:49:00.071400 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:00.070889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wpj5d" event={"ID":"6efe2f4c-63e7-4e75-9bdb-170767d4e22c","Type":"ContainerDied","Data":"e6d98c1fe9ee20339bf7d0803e7e8df0cfa0615cad821309ea9bd9db2ab20779"}
Sep 17 17:49:01.205709 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:01.205687    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wpj5d"
Sep 17 17:49:01.291789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:01.291746    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6vqh\" (UniqueName: \"kubernetes.io/projected/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-kube-api-access-m6vqh\") pod \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") "
Sep 17 17:49:01.291789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:01.291796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-model-cache\") pod \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\" (UID: \"6efe2f4c-63e7-4e75-9bdb-170767d4e22c\") "
Sep 17 17:49:01.293830 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:01.293807    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-kube-api-access-m6vqh" (OuterVolumeSpecName: "kube-api-access-m6vqh") pod "6efe2f4c-63e7-4e75-9bdb-170767d4e22c" (UID: "6efe2f4c-63e7-4e75-9bdb-170767d4e22c"). InnerVolumeSpecName "kube-api-access-m6vqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:01.310795 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:01.310770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6efe2f4c-63e7-4e75-9bdb-170767d4e22c" (UID: "6efe2f4c-63e7-4e75-9bdb-170767d4e22c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:01.392599 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:01.392525    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6vqh\" (UniqueName: \"kubernetes.io/projected/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-kube-api-access-m6vqh\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:49:01.392599 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:01.392551    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe2f4c-63e7-4e75-9bdb-170767d4e22c-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:49:02.080330 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:02.080296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wpj5d" event={"ID":"6efe2f4c-63e7-4e75-9bdb-170767d4e22c","Type":"ContainerDied","Data":"2fae070b2267b2681c5d9cc0479714927f4b627da4f6cf989042b29c44fb87a8"}
Sep 17 17:49:02.080330 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:02.080336    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fae070b2267b2681c5d9cc0479714927f4b627da4f6cf989042b29c44fb87a8"
Sep 17 17:49:02.080330 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:02.080335    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wpj5d"
Sep 17 17:49:02.430123 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:02.430057    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:49:02.430123 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:02.430103    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:49:24.316414 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.316361    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"]
Sep 17 17:49:24.316846 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.316674    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6efe2f4c-63e7-4e75-9bdb-170767d4e22c" containerName="download-hf-model"
Sep 17 17:49:24.316846 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.316685    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6efe2f4c-63e7-4e75-9bdb-170767d4e22c" containerName="download-hf-model"
Sep 17 17:49:24.316846 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.316709    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6efe2f4c-63e7-4e75-9bdb-170767d4e22c" containerName="s3-init"
Sep 17 17:49:24.316846 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.316714    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6efe2f4c-63e7-4e75-9bdb-170767d4e22c" containerName="s3-init"
Sep 17 17:49:24.316846 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.316762    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6efe2f4c-63e7-4e75-9bdb-170767d4e22c" containerName="s3-init"
Sep 17 17:49:24.318703 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.318688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.322416 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.322394    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 17:49:24.322579 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.322422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 17:49:24.322721 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.322434    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 17:49:24.322721 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.322481    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jjghq\""
Sep 17 17:49:24.333523 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.333503    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"]
Sep 17 17:49:24.364315 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.364292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.364437 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.364339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-dshm\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.364437 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.364362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7895642-05d5-4818-a6e7-ff26c33e5897-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.364437 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.364412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-home\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.364537 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.364459    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.364537 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.364489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgjld\" (UniqueName: \"kubernetes.io/projected/a7895642-05d5-4818-a6e7-ff26c33e5897-kube-api-access-qgjld\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.465759 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.465731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-dshm\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.465759 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.465760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7895642-05d5-4818-a6e7-ff26c33e5897-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.465945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.465787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-home\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.465945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.465811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.465945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.465831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjld\" (UniqueName: \"kubernetes.io/projected/a7895642-05d5-4818-a6e7-ff26c33e5897-kube-api-access-qgjld\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.465945 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.465866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.466225 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.466205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-home\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.466290 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.466244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.466341 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.466289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.468004 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.467978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-dshm\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.468288 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.468269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7895642-05d5-4818-a6e7-ff26c33e5897-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.474681 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.474659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjld\" (UniqueName: \"kubernetes.io/projected/a7895642-05d5-4818-a6e7-ff26c33e5897-kube-api-access-qgjld\") pod \"fc-smoke-test-kserve-7f4687897-zvk58\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.628879 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.628810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:24.901852 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.901825    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"]
Sep 17 17:49:24.904059 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:49:24.904027    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7895642_05d5_4818_a6e7_ff26c33e5897.slice/crio-2cfe282ea59291c42b3358894c0774cf6e3be422bfc603a1ab82afe62c78033c WatchSource:0}: Error finding container 2cfe282ea59291c42b3358894c0774cf6e3be422bfc603a1ab82afe62c78033c: Status 404 returned error can't find the container with id 2cfe282ea59291c42b3358894c0774cf6e3be422bfc603a1ab82afe62c78033c
Sep 17 17:49:24.906138 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:24.906119    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:49:25.175597 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:25.175530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" event={"ID":"a7895642-05d5-4818-a6e7-ff26c33e5897","Type":"ContainerStarted","Data":"2cfe282ea59291c42b3358894c0774cf6e3be422bfc603a1ab82afe62c78033c"}
Sep 17 17:49:27.185064 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:27.185025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" event={"ID":"a7895642-05d5-4818-a6e7-ff26c33e5897","Type":"ContainerStarted","Data":"03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479"}
Sep 17 17:49:27.205458 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:27.205412    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" podStartSLOduration=1.78561447 podStartE2EDuration="3.205396188s" totalImagesPullingTime="1.419781718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:27.204291644 +0000 UTC m=+927.063883000" watchObservedRunningTime="2026-09-17 17:49:27.205396188 +0000 UTC m=+927.064987544"
Sep 17 17:49:27.629792 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:27.629766    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:34.629095 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:34.629059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:34.641330 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:34.641306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:35.227397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:35.227324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:49:37.836168 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.836137    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"]
Sep 17 17:49:37.842018 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.840739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:37.844502 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.844468    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 17:49:37.846297 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.845947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 17:49:37.846297 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.846059    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-dwrjn\""
Sep 17 17:49:37.846297 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.846111    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 17:49:37.846297 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.845905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-vfndm\""
Sep 17 17:49:37.855458 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.855429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"]
Sep 17 17:49:37.972846 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.972815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnlcw\" (UniqueName: \"kubernetes.io/projected/b50e9442-d1e6-4857-b8d3-74dcd7552e51-kube-api-access-mnlcw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:37.973058 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:37.972851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50e9442-d1e6-4857-b8d3-74dcd7552e51-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:38.073349 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:38.073307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnlcw\" (UniqueName: \"kubernetes.io/projected/b50e9442-d1e6-4857-b8d3-74dcd7552e51-kube-api-access-mnlcw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:38.073349 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:38.073353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50e9442-d1e6-4857-b8d3-74dcd7552e51-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:38.075949 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:38.075905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50e9442-d1e6-4857-b8d3-74dcd7552e51-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:38.083115 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:38.083090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnlcw\" (UniqueName: \"kubernetes.io/projected/b50e9442-d1e6-4857-b8d3-74dcd7552e51-kube-api-access-mnlcw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:38.153307 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:38.153230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:38.421660 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:38.421637    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"]
Sep 17 17:49:38.423493 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:49:38.423470    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50e9442_d1e6_4857_b8d3_74dcd7552e51.slice/crio-990dcb90c27d9337b0654a19e6fd34afe35972e69498f4716f444797869529a5 WatchSource:0}: Error finding container 990dcb90c27d9337b0654a19e6fd34afe35972e69498f4716f444797869529a5: Status 404 returned error can't find the container with id 990dcb90c27d9337b0654a19e6fd34afe35972e69498f4716f444797869529a5
Sep 17 17:49:39.237454 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:39.237410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" event={"ID":"b50e9442-d1e6-4857-b8d3-74dcd7552e51","Type":"ContainerStarted","Data":"990dcb90c27d9337b0654a19e6fd34afe35972e69498f4716f444797869529a5"}
Sep 17 17:49:41.249249 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:41.249192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" event={"ID":"b50e9442-d1e6-4857-b8d3-74dcd7552e51","Type":"ContainerStarted","Data":"82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441"}
Sep 17 17:49:41.249722 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:41.249273    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:49:41.267133 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:49:41.267090    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" podStartSLOduration=2.200904568 podStartE2EDuration="4.267074929s" totalImagesPullingTime="2.066170361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:41.265750658 +0000 UTC m=+941.125342014" watchObservedRunningTime="2026-09-17 17:49:41.267074929 +0000 UTC m=+941.126666285"
Sep 17 17:50:12.254961 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:12.254931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:50:15.212218 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:15.212142    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"]
Sep 17 17:50:18.247646 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.247613    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"]
Sep 17 17:50:18.256567 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:50:18.256529    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:50:18.256567 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:50:18.256567 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:50:18.256567 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479" cmd=["/bin/sleep","15"]
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.256587    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" message="" err=<
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:50:18.256602    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" podUID="a7895642-05d5-4818-a6e7-ff26c33e5897" containerName="main" containerID="cri-o://03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479"
Sep 17 17:50:18.256838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.256652    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" podUID="a7895642-05d5-4818-a6e7-ff26c33e5897" containerName="main" containerID="cri-o://03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479" gracePeriod=60
Sep 17 17:50:18.599002 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.598977    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:50:18.613177 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-tmp-dir\") pod \"a7895642-05d5-4818-a6e7-ff26c33e5897\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") "
Sep 17 17:50:18.613364 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7895642-05d5-4818-a6e7-ff26c33e5897-tls-certs\") pod \"a7895642-05d5-4818-a6e7-ff26c33e5897\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") "
Sep 17 17:50:18.613364 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613203    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-home\") pod \"a7895642-05d5-4818-a6e7-ff26c33e5897\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") "
Sep 17 17:50:18.613364 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613227    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgjld\" (UniqueName: \"kubernetes.io/projected/a7895642-05d5-4818-a6e7-ff26c33e5897-kube-api-access-qgjld\") pod \"a7895642-05d5-4818-a6e7-ff26c33e5897\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") "
Sep 17 17:50:18.613364 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-model-cache\") pod \"a7895642-05d5-4818-a6e7-ff26c33e5897\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") "
Sep 17 17:50:18.613364 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613296    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-dshm\") pod \"a7895642-05d5-4818-a6e7-ff26c33e5897\" (UID: \"a7895642-05d5-4818-a6e7-ff26c33e5897\") "
Sep 17 17:50:18.613639 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613425    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7895642-05d5-4818-a6e7-ff26c33e5897" (UID: "a7895642-05d5-4818-a6e7-ff26c33e5897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:18.613639 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7895642-05d5-4818-a6e7-ff26c33e5897" (UID: "a7895642-05d5-4818-a6e7-ff26c33e5897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:18.613639 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613602    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:18.613997 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.613973    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-home" (OuterVolumeSpecName: "home") pod "a7895642-05d5-4818-a6e7-ff26c33e5897" (UID: "a7895642-05d5-4818-a6e7-ff26c33e5897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:18.615497 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.615464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7895642-05d5-4818-a6e7-ff26c33e5897-kube-api-access-qgjld" (OuterVolumeSpecName: "kube-api-access-qgjld") pod "a7895642-05d5-4818-a6e7-ff26c33e5897" (UID: "a7895642-05d5-4818-a6e7-ff26c33e5897"). InnerVolumeSpecName "kube-api-access-qgjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:18.615605 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.615492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-dshm" (OuterVolumeSpecName: "dshm") pod "a7895642-05d5-4818-a6e7-ff26c33e5897" (UID: "a7895642-05d5-4818-a6e7-ff26c33e5897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:18.615605 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.615545    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7895642-05d5-4818-a6e7-ff26c33e5897-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7895642-05d5-4818-a6e7-ff26c33e5897" (UID: "a7895642-05d5-4818-a6e7-ff26c33e5897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:18.714845 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.714820    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7895642-05d5-4818-a6e7-ff26c33e5897-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:18.714845 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.714842    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:18.714845 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.714852    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgjld\" (UniqueName: \"kubernetes.io/projected/a7895642-05d5-4818-a6e7-ff26c33e5897-kube-api-access-qgjld\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:18.715067 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.714860    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:18.715067 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:18.714868    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7895642-05d5-4818-a6e7-ff26c33e5897-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:19.407114 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.407080    2566 generic.go:356] "Generic (PLEG): container finished" podID="a7895642-05d5-4818-a6e7-ff26c33e5897" containerID="03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479" exitCode=0
Sep 17 17:50:19.407552 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.407124    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"
Sep 17 17:50:19.407552 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.407164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" event={"ID":"a7895642-05d5-4818-a6e7-ff26c33e5897","Type":"ContainerDied","Data":"03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479"}
Sep 17 17:50:19.407552 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.407216    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58" event={"ID":"a7895642-05d5-4818-a6e7-ff26c33e5897","Type":"ContainerDied","Data":"2cfe282ea59291c42b3358894c0774cf6e3be422bfc603a1ab82afe62c78033c"}
Sep 17 17:50:19.407552 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.407238    2566 scope.go:117] "RemoveContainer" containerID="03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479"
Sep 17 17:50:19.415969 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.415951    2566 scope.go:117] "RemoveContainer" containerID="03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479"
Sep 17 17:50:19.416254 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:50:19.416236    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479\": container with ID starting with 03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479 not found: ID does not exist" containerID="03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479"
Sep 17 17:50:19.416336 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.416266    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479"} err="failed to get container status \"03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479\": rpc error: code = NotFound desc = could not find container \"03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479\": container with ID starting with 03afc74688d074ca0e2e73a954e4f3f092b50710059ed75749c13560e1f98479 not found: ID does not exist"
Sep 17 17:50:19.429621 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.429599    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"]
Sep 17 17:50:19.434670 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:19.434652    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-zvk58"]
Sep 17 17:50:20.506227 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:20.506191    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7895642-05d5-4818-a6e7-ff26c33e5897" path="/var/lib/kubelet/pods/a7895642-05d5-4818-a6e7-ff26c33e5897/volumes"
Sep 17 17:50:22.254599 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:22.254563    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" podUID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:26.792252 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.792204    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 17:50:26.792252 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:50:26.792244    2566 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" podUID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" containerName="main" containerID="cri-o://82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441"
Sep 17 17:50:26.792597 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.792292    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" podUID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" containerName="main" containerID="cri-o://82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441" gracePeriod=49
Sep 17 17:50:26.913961 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.913938    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp_b50e9442-d1e6-4857-b8d3-74dcd7552e51/main/0.log"
Sep 17 17:50:26.916274 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.916254    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:50:26.978673 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.978647    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50e9442-d1e6-4857-b8d3-74dcd7552e51-tls-certs\") pod \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") "
Sep 17 17:50:26.978795 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.978685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnlcw\" (UniqueName: \"kubernetes.io/projected/b50e9442-d1e6-4857-b8d3-74dcd7552e51-kube-api-access-mnlcw\") pod \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\" (UID: \"b50e9442-d1e6-4857-b8d3-74dcd7552e51\") "
Sep 17 17:50:26.980679 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.980654    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50e9442-d1e6-4857-b8d3-74dcd7552e51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b50e9442-d1e6-4857-b8d3-74dcd7552e51" (UID: "b50e9442-d1e6-4857-b8d3-74dcd7552e51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:26.980757 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:26.980735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50e9442-d1e6-4857-b8d3-74dcd7552e51-kube-api-access-mnlcw" (OuterVolumeSpecName: "kube-api-access-mnlcw") pod "b50e9442-d1e6-4857-b8d3-74dcd7552e51" (UID: "b50e9442-d1e6-4857-b8d3-74dcd7552e51"). InnerVolumeSpecName "kube-api-access-mnlcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:27.080227 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.080191    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50e9442-d1e6-4857-b8d3-74dcd7552e51-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:27.080227 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.080223    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnlcw\" (UniqueName: \"kubernetes.io/projected/b50e9442-d1e6-4857-b8d3-74dcd7552e51-kube-api-access-mnlcw\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:50:27.443331 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.443259    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp_b50e9442-d1e6-4857-b8d3-74dcd7552e51/main/0.log"
Sep 17 17:50:27.443331 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.443299    2566 generic.go:356] "Generic (PLEG): container finished" podID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" containerID="82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441" exitCode=1
Sep 17 17:50:27.443498 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.443346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" event={"ID":"b50e9442-d1e6-4857-b8d3-74dcd7552e51","Type":"ContainerDied","Data":"82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441"}
Sep 17 17:50:27.443498 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.443389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp" event={"ID":"b50e9442-d1e6-4857-b8d3-74dcd7552e51","Type":"ContainerDied","Data":"990dcb90c27d9337b0654a19e6fd34afe35972e69498f4716f444797869529a5"}
Sep 17 17:50:27.443498 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.443411    2566 scope.go:117] "RemoveContainer" containerID="82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441"
Sep 17 17:50:27.443498 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.443348    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"
Sep 17 17:50:27.452691 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.452663    2566 scope.go:117] "RemoveContainer" containerID="82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441"
Sep 17 17:50:27.452974 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:50:27.452956    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441\": container with ID starting with 82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441 not found: ID does not exist" containerID="82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441"
Sep 17 17:50:27.453054 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.452984    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441"} err="failed to get container status \"82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441\": rpc error: code = NotFound desc = could not find container \"82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441\": container with ID starting with 82e2547fb824681113b48e204bc5883eb803e75a01e2a70d3d0f8483f596f441 not found: ID does not exist"
Sep 17 17:50:27.465972 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.465950    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"]
Sep 17 17:50:27.470145 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:27.470126    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wlwsp"]
Sep 17 17:50:28.506453 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:50:28.506426    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" path="/var/lib/kubelet/pods/b50e9442-d1e6-4857-b8d3-74dcd7552e51/volumes"
Sep 17 17:51:12.704250 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.704213    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"]
Sep 17 17:51:12.705016 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.704685    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7895642-05d5-4818-a6e7-ff26c33e5897" containerName="main"
Sep 17 17:51:12.705016 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.704703    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7895642-05d5-4818-a6e7-ff26c33e5897" containerName="main"
Sep 17 17:51:12.705016 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.704716    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" containerName="main"
Sep 17 17:51:12.705016 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.704723    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" containerName="main"
Sep 17 17:51:12.705016 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.704790    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50e9442-d1e6-4857-b8d3-74dcd7552e51" containerName="main"
Sep 17 17:51:12.705016 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.704804    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7895642-05d5-4818-a6e7-ff26c33e5897" containerName="main"
Sep 17 17:51:12.708802 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.708783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.712255 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.712235    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 17:51:12.713137 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.713106    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-8dqrz\""
Sep 17 17:51:12.713335 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.713322    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-sxn6v\""
Sep 17 17:51:12.713444 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.713360    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 17:51:12.713444 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.713363    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 17:51:12.722576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.722557    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"]
Sep 17 17:51:12.744427 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.744397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.744519 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.744442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e65e0f-529f-44c1-a32d-de9d62f58aba-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.744563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.744516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.744563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.744547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnt4\" (UniqueName: \"kubernetes.io/projected/86e65e0f-529f-44c1-a32d-de9d62f58aba-kube-api-access-wgnt4\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.845491 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.845454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.845652 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.845506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e65e0f-529f-44c1-a32d-de9d62f58aba-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.845652 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.845536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.845652 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.845573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnt4\" (UniqueName: \"kubernetes.io/projected/86e65e0f-529f-44c1-a32d-de9d62f58aba-kube-api-access-wgnt4\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.845935 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.845887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.845984 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.845949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.847934 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.847899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e65e0f-529f-44c1-a32d-de9d62f58aba-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:12.855237 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:12.855214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnt4\" (UniqueName: \"kubernetes.io/projected/86e65e0f-529f-44c1-a32d-de9d62f58aba-kube-api-access-wgnt4\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:13.018405 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:13.018324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:13.297757 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:13.297731    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"]
Sep 17 17:51:13.301989 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:51:13.301964    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e65e0f_529f_44c1_a32d_de9d62f58aba.slice/crio-4f095c8495eac7168ee6fe7bfb2322e47b13e62ff2befcc53cc21848a5139401 WatchSource:0}: Error finding container 4f095c8495eac7168ee6fe7bfb2322e47b13e62ff2befcc53cc21848a5139401: Status 404 returned error can't find the container with id 4f095c8495eac7168ee6fe7bfb2322e47b13e62ff2befcc53cc21848a5139401
Sep 17 17:51:13.631021 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:13.630985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" event={"ID":"86e65e0f-529f-44c1-a32d-de9d62f58aba","Type":"ContainerStarted","Data":"9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2"}
Sep 17 17:51:13.631201 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:13.631037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" event={"ID":"86e65e0f-529f-44c1-a32d-de9d62f58aba","Type":"ContainerStarted","Data":"4f095c8495eac7168ee6fe7bfb2322e47b13e62ff2befcc53cc21848a5139401"}
Sep 17 17:51:13.631201 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:13.631083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:13.653314 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:13.653263    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" podStartSLOduration=1.6532443780000001 podStartE2EDuration="1.653244378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:13.65184228 +0000 UTC m=+1033.511433635" watchObservedRunningTime="2026-09-17 17:51:13.653244378 +0000 UTC m=+1033.512835733"
Sep 17 17:51:26.728303 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.728271    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"]
Sep 17 17:51:26.731677 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.731662    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.734182 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.734160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-ct45r\""
Sep 17 17:51:26.735243 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.735221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 17:51:26.735354 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.735240    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 17:51:26.735354 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.735223    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 17:51:26.742673 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.742652    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"]
Sep 17 17:51:26.756367 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.756346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.756468 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.756381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.756468 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.756409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-home\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.756468 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.756451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.756580 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.756529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdl6\" (UniqueName: \"kubernetes.io/projected/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-kube-api-access-ccdl6\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.756580 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.756567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.857849 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.857825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-home\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.857965 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.857851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.857965 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.857884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdl6\" (UniqueName: \"kubernetes.io/projected/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-kube-api-access-ccdl6\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.857965 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.857903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.858120 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.857966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.858120 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.857990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.858302 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.858282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-home\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.858302 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.858294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.858401 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.858356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.860160 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.860138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.860413 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.860396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:26.867463 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:26.867445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdl6\" (UniqueName: \"kubernetes.io/projected/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-kube-api-access-ccdl6\") pod \"fc-concurrency-test-kserve-b5d44467c-mq85l\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:27.042372 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:27.042294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:27.311215 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:27.311193    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"]
Sep 17 17:51:27.313225 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:51:27.313199    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfb31b9_1bea_4aad_b1f9_11a47aa41559.slice/crio-ccb49a77fd7bcb81af813f25fcabd7bf65fe396c5b435ebc564b2afa733467cc WatchSource:0}: Error finding container ccb49a77fd7bcb81af813f25fcabd7bf65fe396c5b435ebc564b2afa733467cc: Status 404 returned error can't find the container with id ccb49a77fd7bcb81af813f25fcabd7bf65fe396c5b435ebc564b2afa733467cc
Sep 17 17:51:27.691997 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:27.691900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" event={"ID":"4bfb31b9-1bea-4aad-b1f9-11a47aa41559","Type":"ContainerStarted","Data":"9f5f65f9bdaafc3794e8fa4b5486c5f214279cbba79654ae3a35b15e91e93c84"}
Sep 17 17:51:27.691997 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:27.691969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" event={"ID":"4bfb31b9-1bea-4aad-b1f9-11a47aa41559","Type":"ContainerStarted","Data":"ccb49a77fd7bcb81af813f25fcabd7bf65fe396c5b435ebc564b2afa733467cc"}
Sep 17 17:51:28.042848 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:28.042770    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:37.042844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:37.042808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:37.054624 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:37.054591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:37.077789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:37.077746    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" podStartSLOduration=11.077733807 podStartE2EDuration="11.077733807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:27.714012941 +0000 UTC m=+1047.573604296" watchObservedRunningTime="2026-09-17 17:51:37.077733807 +0000 UTC m=+1056.937325162"
Sep 17 17:51:37.741388 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:37.741362    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:51:44.636267 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:44.636195    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:51:51.384324 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:51.384294    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"]
Sep 17 17:51:54.636533 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:51:54.636499    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:04.637099 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:04.637062    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:06.403248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.403209    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main" containerID="cri-o://9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2" gracePeriod=45
Sep 17 17:52:06.548511 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.548477    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"]
Sep 17 17:52:06.557032 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:52:06.556997    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:52:06.557032 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:52:06.557032 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:52:06.557032 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="9f5f65f9bdaafc3794e8fa4b5486c5f214279cbba79654ae3a35b15e91e93c84" cmd=["/bin/sleep","15"]
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.557045    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" message="" err=<
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:52:06.557063    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" podUID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" containerName="main" containerID="cri-o://9f5f65f9bdaafc3794e8fa4b5486c5f214279cbba79654ae3a35b15e91e93c84"
Sep 17 17:52:06.557333 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.557101    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" podUID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" containerName="main" containerID="cri-o://9f5f65f9bdaafc3794e8fa4b5486c5f214279cbba79654ae3a35b15e91e93c84" gracePeriod=60
Sep 17 17:52:06.731234 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.731201    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" podUID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 17:52:06.849872 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.849843    2566 generic.go:356] "Generic (PLEG): container finished" podID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" containerID="9f5f65f9bdaafc3794e8fa4b5486c5f214279cbba79654ae3a35b15e91e93c84" exitCode=0
Sep 17 17:52:06.850036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.849934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" event={"ID":"4bfb31b9-1bea-4aad-b1f9-11a47aa41559","Type":"ContainerDied","Data":"9f5f65f9bdaafc3794e8fa4b5486c5f214279cbba79654ae3a35b15e91e93c84"}
Sep 17 17:52:06.897451 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.897430    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:52:06.992156 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992095    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccdl6\" (UniqueName: \"kubernetes.io/projected/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-kube-api-access-ccdl6\") pod \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") "
Sep 17 17:52:06.992276 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tmp-dir\") pod \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") "
Sep 17 17:52:06.992276 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-model-cache\") pod \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") "
Sep 17 17:52:06.992276 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992267    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tls-certs\") pod \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") "
Sep 17 17:52:06.992412 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992298    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-dshm\") pod \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") "
Sep 17 17:52:06.992412 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992321    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-home\") pod \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\" (UID: \"4bfb31b9-1bea-4aad-b1f9-11a47aa41559\") "
Sep 17 17:52:06.992510 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992458    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bfb31b9-1bea-4aad-b1f9-11a47aa41559" (UID: "4bfb31b9-1bea-4aad-b1f9-11a47aa41559"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:06.992564 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992506    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bfb31b9-1bea-4aad-b1f9-11a47aa41559" (UID: "4bfb31b9-1bea-4aad-b1f9-11a47aa41559"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:06.992627 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992559    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-home" (OuterVolumeSpecName: "home") pod "4bfb31b9-1bea-4aad-b1f9-11a47aa41559" (UID: "4bfb31b9-1bea-4aad-b1f9-11a47aa41559"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:06.992677 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992642    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:06.992677 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992664    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:06.992742 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.992677    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:06.994232 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.994197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-kube-api-access-ccdl6" (OuterVolumeSpecName: "kube-api-access-ccdl6") pod "4bfb31b9-1bea-4aad-b1f9-11a47aa41559" (UID: "4bfb31b9-1bea-4aad-b1f9-11a47aa41559"). InnerVolumeSpecName "kube-api-access-ccdl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:06.994342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.994276    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-dshm" (OuterVolumeSpecName: "dshm") pod "4bfb31b9-1bea-4aad-b1f9-11a47aa41559" (UID: "4bfb31b9-1bea-4aad-b1f9-11a47aa41559"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:06.994342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:06.994303    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bfb31b9-1bea-4aad-b1f9-11a47aa41559" (UID: "4bfb31b9-1bea-4aad-b1f9-11a47aa41559"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:07.093047 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.093021    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:07.093047 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.093041    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:07.093047 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.093050    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdl6\" (UniqueName: \"kubernetes.io/projected/4bfb31b9-1bea-4aad-b1f9-11a47aa41559-kube-api-access-ccdl6\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:07.854598 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.854573    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"
Sep 17 17:52:07.855042 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.854590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l" event={"ID":"4bfb31b9-1bea-4aad-b1f9-11a47aa41559","Type":"ContainerDied","Data":"ccb49a77fd7bcb81af813f25fcabd7bf65fe396c5b435ebc564b2afa733467cc"}
Sep 17 17:52:07.855042 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.854649    2566 scope.go:117] "RemoveContainer" containerID="9f5f65f9bdaafc3794e8fa4b5486c5f214279cbba79654ae3a35b15e91e93c84"
Sep 17 17:52:07.881704 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.881677    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"]
Sep 17 17:52:07.892462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:07.892440    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-mq85l"]
Sep 17 17:52:08.506452 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:08.506420    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" path="/var/lib/kubelet/pods/4bfb31b9-1bea-4aad-b1f9-11a47aa41559/volumes"
Sep 17 17:52:14.636428 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:14.636392    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:14.636778 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:14.636485    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:52:24.636755 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:24.636720    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:34.636466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:34.636433    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:36.648306 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.648283    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:52:36.705786 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.705719    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-tokenizer-cache\") pod \"86e65e0f-529f-44c1-a32d-de9d62f58aba\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") "
Sep 17 17:52:36.705786 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.705772    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgnt4\" (UniqueName: \"kubernetes.io/projected/86e65e0f-529f-44c1-a32d-de9d62f58aba-kube-api-access-wgnt4\") pod \"86e65e0f-529f-44c1-a32d-de9d62f58aba\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") "
Sep 17 17:52:36.706015 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.705805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e65e0f-529f-44c1-a32d-de9d62f58aba-tls-certs\") pod \"86e65e0f-529f-44c1-a32d-de9d62f58aba\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") "
Sep 17 17:52:36.706015 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.705832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-cachi2-cache\") pod \"86e65e0f-529f-44c1-a32d-de9d62f58aba\" (UID: \"86e65e0f-529f-44c1-a32d-de9d62f58aba\") "
Sep 17 17:52:36.706127 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.706035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "86e65e0f-529f-44c1-a32d-de9d62f58aba" (UID: "86e65e0f-529f-44c1-a32d-de9d62f58aba"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:36.706198 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.706167    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-tokenizer-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:36.706253 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.706190    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "86e65e0f-529f-44c1-a32d-de9d62f58aba" (UID: "86e65e0f-529f-44c1-a32d-de9d62f58aba"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:36.708069 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.708040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e65e0f-529f-44c1-a32d-de9d62f58aba-kube-api-access-wgnt4" (OuterVolumeSpecName: "kube-api-access-wgnt4") pod "86e65e0f-529f-44c1-a32d-de9d62f58aba" (UID: "86e65e0f-529f-44c1-a32d-de9d62f58aba"). InnerVolumeSpecName "kube-api-access-wgnt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:36.708069 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.708063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e65e0f-529f-44c1-a32d-de9d62f58aba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86e65e0f-529f-44c1-a32d-de9d62f58aba" (UID: "86e65e0f-529f-44c1-a32d-de9d62f58aba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:36.806843 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.806810    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgnt4\" (UniqueName: \"kubernetes.io/projected/86e65e0f-529f-44c1-a32d-de9d62f58aba-kube-api-access-wgnt4\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:36.806843 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.806838    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e65e0f-529f-44c1-a32d-de9d62f58aba-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:36.807044 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.806852    2566 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e65e0f-529f-44c1-a32d-de9d62f58aba-cachi2-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:52:36.970553 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.970469    2566 generic.go:356] "Generic (PLEG): container finished" podID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerID="9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2" exitCode=0
Sep 17 17:52:36.970553 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.970520    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"
Sep 17 17:52:36.970553 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.970524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" event={"ID":"86e65e0f-529f-44c1-a32d-de9d62f58aba","Type":"ContainerDied","Data":"9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2"}
Sep 17 17:52:36.970828 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.970576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v" event={"ID":"86e65e0f-529f-44c1-a32d-de9d62f58aba","Type":"ContainerDied","Data":"4f095c8495eac7168ee6fe7bfb2322e47b13e62ff2befcc53cc21848a5139401"}
Sep 17 17:52:36.970828 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.970604    2566 scope.go:117] "RemoveContainer" containerID="9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2"
Sep 17 17:52:36.982397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.981173    2566 scope.go:117] "RemoveContainer" containerID="9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2"
Sep 17 17:52:36.982397 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:52:36.981525    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2\": container with ID starting with 9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2 not found: ID does not exist" containerID="9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2"
Sep 17 17:52:36.982397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:36.981564    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2"} err="failed to get container status \"9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2\": rpc error: code = NotFound desc = could not find container \"9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2\": container with ID starting with 9adc12723e8daedce4a71b057e191f3bd6c1efdbfb300c2f8f4b24ed4c5280a2 not found: ID does not exist"
Sep 17 17:52:37.012282 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:37.012257    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"]
Sep 17 17:52:37.018841 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:37.018817    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc4m82v"]
Sep 17 17:52:38.506228 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:38.506192    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" path="/var/lib/kubelet/pods/86e65e0f-529f-44c1-a32d-de9d62f58aba/volumes"
Sep 17 17:52:50.954680 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.954649    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"]
Sep 17 17:52:50.955160 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.955007    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" containerName="main"
Sep 17 17:52:50.955160 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.955019    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" containerName="main"
Sep 17 17:52:50.955160 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.955032    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main"
Sep 17 17:52:50.955160 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.955037    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main"
Sep 17 17:52:50.955160 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.955086    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bfb31b9-1bea-4aad-b1f9-11a47aa41559" containerName="main"
Sep 17 17:52:50.955160 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.955096    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e65e0f-529f-44c1-a32d-de9d62f58aba" containerName="main"
Sep 17 17:52:50.957995 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.957973    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:50.963168 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.963135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-nd4c6\""
Sep 17 17:52:50.963340 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.963222    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 17:52:50.964098 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.964075    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 17:52:50.964547 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.964529    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 17:52:50.964740 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.964725    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-cm9mx\""
Sep 17 17:52:50.978636 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:50.978605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"]
Sep 17 17:52:51.029011 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.028978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sls9\" (UniqueName: \"kubernetes.io/projected/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-kube-api-access-6sls9\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.029199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.029025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.029199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.029083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.029199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.029120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.029199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.029151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.029199 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.029184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.129742 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.129705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.129950 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.129752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.129950 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.129785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.129950 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.129817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.129950 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.129856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.129950 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.129903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sls9\" (UniqueName: \"kubernetes.io/projected/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-kube-api-access-6sls9\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.130224 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.130206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.130286 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.130239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.130332 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.130280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.132035 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.132016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.132353 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.132337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.140228 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.140209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sls9\" (UniqueName: \"kubernetes.io/projected/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-kube-api-access-6sls9\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.222877 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.222786    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"]
Sep 17 17:52:51.226564 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.226542    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.229821 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.229799    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-sb9bq\""
Sep 17 17:52:51.234776 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.234757    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"]
Sep 17 17:52:51.268511 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.268472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:51.331718 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.331688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv9nt\" (UniqueName: \"kubernetes.io/projected/aa8b0cf2-1545-430d-af58-75c84b08d3cb-kube-api-access-rv9nt\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.331841 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.331746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8b0cf2-1545-430d-af58-75c84b08d3cb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.433289 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.433252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv9nt\" (UniqueName: \"kubernetes.io/projected/aa8b0cf2-1545-430d-af58-75c84b08d3cb-kube-api-access-rv9nt\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.433479 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.433308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8b0cf2-1545-430d-af58-75c84b08d3cb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.435775 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.435756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8b0cf2-1545-430d-af58-75c84b08d3cb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.443323 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.443291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv9nt\" (UniqueName: \"kubernetes.io/projected/aa8b0cf2-1545-430d-af58-75c84b08d3cb-kube-api-access-rv9nt\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.537900 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.537820    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:51.538946 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.538865    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"]
Sep 17 17:52:51.541828 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:52:51.541798    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd29c8c_ce07_452e_a9d3_ba30adba0fb4.slice/crio-3a4d106093f8050fb26f22a44985e1b559d565c789dd6033e31bd38861fa31bb WatchSource:0}: Error finding container 3a4d106093f8050fb26f22a44985e1b559d565c789dd6033e31bd38861fa31bb: Status 404 returned error can't find the container with id 3a4d106093f8050fb26f22a44985e1b559d565c789dd6033e31bd38861fa31bb
Sep 17 17:52:51.815104 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:51.815079    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"]
Sep 17 17:52:51.818708 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:52:51.818678    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8b0cf2_1545_430d_af58_75c84b08d3cb.slice/crio-93ec1296c93150576ef30c707a9ba1410ed6536f281b82e59afc4a82d19d7af4 WatchSource:0}: Error finding container 93ec1296c93150576ef30c707a9ba1410ed6536f281b82e59afc4a82d19d7af4: Status 404 returned error can't find the container with id 93ec1296c93150576ef30c707a9ba1410ed6536f281b82e59afc4a82d19d7af4
Sep 17 17:52:52.034478 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:52.034441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" event={"ID":"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4","Type":"ContainerStarted","Data":"3a4d106093f8050fb26f22a44985e1b559d565c789dd6033e31bd38861fa31bb"}
Sep 17 17:52:52.035835 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:52.035803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" event={"ID":"aa8b0cf2-1545-430d-af58-75c84b08d3cb","Type":"ContainerStarted","Data":"5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a"}
Sep 17 17:52:52.035957 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:52.035854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" event={"ID":"aa8b0cf2-1545-430d-af58-75c84b08d3cb","Type":"ContainerStarted","Data":"93ec1296c93150576ef30c707a9ba1410ed6536f281b82e59afc4a82d19d7af4"}
Sep 17 17:52:52.035957 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:52.035892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:52:52.058220 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:52.058174    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" podStartSLOduration=1.058158726 podStartE2EDuration="1.058158726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:52.056779433 +0000 UTC m=+1131.916370789" watchObservedRunningTime="2026-09-17 17:52:52.058158726 +0000 UTC m=+1131.917750082"
Sep 17 17:52:54.049007 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:54.048971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" event={"ID":"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4","Type":"ContainerStarted","Data":"3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17"}
Sep 17 17:52:54.049402 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:54.049056    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:52:55.054851 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:55.054813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" event={"ID":"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4","Type":"ContainerStarted","Data":"5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376"}
Sep 17 17:52:55.073760 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:55.073670    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" podStartSLOduration=3.52460557 podStartE2EDuration="5.073653673s" totalImagesPullingTime="1.549048103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:55.072350611 +0000 UTC m=+1134.931941966" watchObservedRunningTime="2026-09-17 17:52:55.073653673 +0000 UTC m=+1134.933245029"
Sep 17 17:52:55.269284 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:52:55.269237    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:53:01.269001 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:53:01.268964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:53:01.281579 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:53:01.281555    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:53:02.096736 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:53:02.096708    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:53:03.101503 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:53:03.101473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:53:23.044446 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:53:23.044413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:54:02.462127 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:02.462097    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:54:02.463613 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:02.463587    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:54:38.780956 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:38.780923    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"]
Sep 17 17:54:38.790376 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:54:38.790308    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:54:38.790376 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.790376 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.790376 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376" cmd=["/bin/sleep","15"]
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:38.790451    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" message="" err=<
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:54:38.790472    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="main" containerID="cri-o://5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376"
Sep 17 17:54:38.790734 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:38.790510    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="main" containerID="cri-o://5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376" gracePeriod=60
Sep 17 17:54:38.793304 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:38.793281    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"]
Sep 17 17:54:39.086651 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.086619    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8001/health\": dial tcp 10.134.0.34:8001: connect: connection refused"
Sep 17 17:54:39.406817 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.404873    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="llm-d-routing-sidecar" containerID="cri-o://3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17" gracePeriod=60
Sep 17 17:54:39.503001 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.502971    2566 generic.go:356] "Generic (PLEG): container finished" podID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerID="5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376" exitCode=0
Sep 17 17:54:39.503140 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.503039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" event={"ID":"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4","Type":"ContainerDied","Data":"5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376"}
Sep 17 17:54:39.643156 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.643126    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:54:39.779999 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.779925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tls-certs\") pod \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") "
Sep 17 17:54:39.780135 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780003    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-model-cache\") pod \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") "
Sep 17 17:54:39.780135 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tmp-dir\") pod \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") "
Sep 17 17:54:39.780135 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sls9\" (UniqueName: \"kubernetes.io/projected/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-kube-api-access-6sls9\") pod \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") "
Sep 17 17:54:39.780288 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-dshm\") pod \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") "
Sep 17 17:54:39.780288 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780171    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-home\") pod \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\" (UID: \"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4\") "
Sep 17 17:54:39.780402 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780293    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" (UID: "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.780402 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" (UID: "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.780510 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780484    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.780510 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780502    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.780587 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.780497    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-home" (OuterVolumeSpecName: "home") pod "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" (UID: "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.783537 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.783504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-dshm" (OuterVolumeSpecName: "dshm") pod "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" (UID: "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.788187 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.788161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" (UID: "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:39.788323 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.788222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-kube-api-access-6sls9" (OuterVolumeSpecName: "kube-api-access-6sls9") pod "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" (UID: "bdd29c8c-ce07-452e-a9d3-ba30adba0fb4"). InnerVolumeSpecName "kube-api-access-6sls9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:39.881715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.881687    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sls9\" (UniqueName: \"kubernetes.io/projected/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-kube-api-access-6sls9\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.881715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.881712    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.881867 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.881722    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.881867 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:39.881730    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:54:40.508440 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.508405    2566 generic.go:356] "Generic (PLEG): container finished" podID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerID="3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17" exitCode=0
Sep 17 17:54:40.508616 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.508490    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"
Sep 17 17:54:40.508616 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.508493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" event={"ID":"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4","Type":"ContainerDied","Data":"3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17"}
Sep 17 17:54:40.508616 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.508550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2" event={"ID":"bdd29c8c-ce07-452e-a9d3-ba30adba0fb4","Type":"ContainerDied","Data":"3a4d106093f8050fb26f22a44985e1b559d565c789dd6033e31bd38861fa31bb"}
Sep 17 17:54:40.508616 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.508575    2566 scope.go:117] "RemoveContainer" containerID="5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376"
Sep 17 17:54:40.518265 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.518248    2566 scope.go:117] "RemoveContainer" containerID="3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17"
Sep 17 17:54:40.525791 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.525771    2566 scope.go:117] "RemoveContainer" containerID="5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376"
Sep 17 17:54:40.526053 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:54:40.526034    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376\": container with ID starting with 5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376 not found: ID does not exist" containerID="5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376"
Sep 17 17:54:40.526136 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.526061    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376"} err="failed to get container status \"5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376\": rpc error: code = NotFound desc = could not find container \"5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376\": container with ID starting with 5b6195fd0a6ec49acdfc154c2eefbfbe1df14cc938e5434c6f48b5e603d16376 not found: ID does not exist"
Sep 17 17:54:40.526136 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.526077    2566 scope.go:117] "RemoveContainer" containerID="3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17"
Sep 17 17:54:40.526345 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:54:40.526329    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17\": container with ID starting with 3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17 not found: ID does not exist" containerID="3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17"
Sep 17 17:54:40.526393 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.526352    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17"} err="failed to get container status \"3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17\": rpc error: code = NotFound desc = could not find container \"3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17\": container with ID starting with 3d33841da86ce6b120d32157dd49be27243a86733d17385c798f592cebe2ab17 not found: ID does not exist"
Sep 17 17:54:40.532474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.532452    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"]
Sep 17 17:54:40.539785 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:40.539760    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-7bxl2"]
Sep 17 17:54:42.507075 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:42.507043    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" path="/var/lib/kubelet/pods/bdd29c8c-ce07-452e-a9d3-ba30adba0fb4/volumes"
Sep 17 17:54:43.044714 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:43.044673    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:53.044509 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:53.044470    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:53.812314 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:54:53.812255    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main" containerID="cri-o://5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a" gracePeriod=45
Sep 17 17:55:03.044461 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:03.044427    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:55:03.044893 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:03.044533    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:55:13.044683 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:13.044644    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:55:23.044527 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:23.044491    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:55:24.050468 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.050444    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:55:24.144888 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.144820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8b0cf2-1545-430d-af58-75c84b08d3cb-tls-certs\") pod \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") "
Sep 17 17:55:24.144888 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.144851    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv9nt\" (UniqueName: \"kubernetes.io/projected/aa8b0cf2-1545-430d-af58-75c84b08d3cb-kube-api-access-rv9nt\") pod \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\" (UID: \"aa8b0cf2-1545-430d-af58-75c84b08d3cb\") "
Sep 17 17:55:24.146879 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.146853    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8b0cf2-1545-430d-af58-75c84b08d3cb-kube-api-access-rv9nt" (OuterVolumeSpecName: "kube-api-access-rv9nt") pod "aa8b0cf2-1545-430d-af58-75c84b08d3cb" (UID: "aa8b0cf2-1545-430d-af58-75c84b08d3cb"). InnerVolumeSpecName "kube-api-access-rv9nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:24.147001 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.146869    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa8b0cf2-1545-430d-af58-75c84b08d3cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa8b0cf2-1545-430d-af58-75c84b08d3cb" (UID: "aa8b0cf2-1545-430d-af58-75c84b08d3cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:24.245656 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.245626    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8b0cf2-1545-430d-af58-75c84b08d3cb-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:55:24.245656 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.245649    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv9nt\" (UniqueName: \"kubernetes.io/projected/aa8b0cf2-1545-430d-af58-75c84b08d3cb-kube-api-access-rv9nt\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:55:24.679963 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.679927    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerID="5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a" exitCode=0
Sep 17 17:55:24.680131 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.679974    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"
Sep 17 17:55:24.680131 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.680014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" event={"ID":"aa8b0cf2-1545-430d-af58-75c84b08d3cb","Type":"ContainerDied","Data":"5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a"}
Sep 17 17:55:24.680131 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.680067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr" event={"ID":"aa8b0cf2-1545-430d-af58-75c84b08d3cb","Type":"ContainerDied","Data":"93ec1296c93150576ef30c707a9ba1410ed6536f281b82e59afc4a82d19d7af4"}
Sep 17 17:55:24.680131 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.680088    2566 scope.go:117] "RemoveContainer" containerID="5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a"
Sep 17 17:55:24.688035 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.688012    2566 scope.go:117] "RemoveContainer" containerID="5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a"
Sep 17 17:55:24.688296 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:55:24.688279    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a\": container with ID starting with 5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a not found: ID does not exist" containerID="5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a"
Sep 17 17:55:24.688338 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.688305    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a"} err="failed to get container status \"5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a\": rpc error: code = NotFound desc = could not find container \"5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a\": container with ID starting with 5f3043f5df137e906cd5499d6dc6fa62c4b49ba59dcdba9024abcc1ad8f7e69a not found: ID does not exist"
Sep 17 17:55:24.700658 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.700634    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"]
Sep 17 17:55:24.705695 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:24.705676    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9ndr"]
Sep 17 17:55:26.506181 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:26.506147    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" path="/var/lib/kubelet/pods/aa8b0cf2-1545-430d-af58-75c84b08d3cb/volumes"
Sep 17 17:55:31.658775 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.658738    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"]
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659077    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659092    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659099    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="main"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659105    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="main"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659119    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="llm-d-routing-sidecar"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659125    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="llm-d-routing-sidecar"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659194    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa8b0cf2-1545-430d-af58-75c84b08d3cb" containerName="main"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659204    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="llm-d-routing-sidecar"
Sep 17 17:55:31.659248 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.659215    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd29c8c-ce07-452e-a9d3-ba30adba0fb4" containerName="main"
Sep 17 17:55:31.663872 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.663854    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.666564 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.666545    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 17:55:31.666655 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.666579    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 17:55:31.668114 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.668096    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-v9mch\""
Sep 17 17:55:31.668280 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.668260    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-lkgrt\""
Sep 17 17:55:31.668358 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.668341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 17:55:31.674411 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.674390    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"]
Sep 17 17:55:31.704768 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.704745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.704901 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.704786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.704901 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.704831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfnn\" (UniqueName: \"kubernetes.io/projected/ba108a53-9870-44a8-9aa9-b9a30e6c9851-kube-api-access-xmfnn\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.704901 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.704849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.705090 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.704946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.705090 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.704981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.805524 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.805493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfnn\" (UniqueName: \"kubernetes.io/projected/ba108a53-9870-44a8-9aa9-b9a30e6c9851-kube-api-access-xmfnn\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.805649 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.805532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.805649 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.805560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.805649 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.805596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.805649 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.805637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.805810 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.805683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.805993 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.805972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.806049 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.806030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.806086 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.806068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.807845 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.807823    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.808046 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.808030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.817056 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.817034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfnn\" (UniqueName: \"kubernetes.io/projected/ba108a53-9870-44a8-9aa9-b9a30e6c9851-kube-api-access-xmfnn\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:31.944744 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.944663    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"]
Sep 17 17:55:31.948115 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.948093    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:31.950800 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.950774    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-hcvcl\""
Sep 17 17:55:31.957399 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.957376    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"]
Sep 17 17:55:31.973920 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:31.973887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:32.008019 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.007984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6jz\" (UniqueName: \"kubernetes.io/projected/e51c716d-d95b-456e-859c-7f4babeeb36a-kube-api-access-qc6jz\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsf626\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.008167 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.008079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c716d-d95b-456e-859c-7f4babeeb36a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsf626\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.109247 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.109212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6jz\" (UniqueName: \"kubernetes.io/projected/e51c716d-d95b-456e-859c-7f4babeeb36a-kube-api-access-qc6jz\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsf626\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.109392 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.109274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c716d-d95b-456e-859c-7f4babeeb36a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsf626\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.111633 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.111607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c716d-d95b-456e-859c-7f4babeeb36a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsf626\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.122398 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.122373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6jz\" (UniqueName: \"kubernetes.io/projected/e51c716d-d95b-456e-859c-7f4babeeb36a-kube-api-access-qc6jz\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsf626\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.248139 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.248111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"]
Sep 17 17:55:32.250497 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:55:32.250475    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba108a53_9870_44a8_9aa9_b9a30e6c9851.slice/crio-bf25fa86b79799b78c1ae23d08e165dee34f2023aef3a55b2c885a2be88337c8 WatchSource:0}: Error finding container bf25fa86b79799b78c1ae23d08e165dee34f2023aef3a55b2c885a2be88337c8: Status 404 returned error can't find the container with id bf25fa86b79799b78c1ae23d08e165dee34f2023aef3a55b2c885a2be88337c8
Sep 17 17:55:32.258009 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.257993    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.530031 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.529927    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"]
Sep 17 17:55:32.532408 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:55:32.532381    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51c716d_d95b_456e_859c_7f4babeeb36a.slice/crio-1fb8c86f100a10467d5f6faf7983505b8df8a0603e67fffc6ee732b41b33155b WatchSource:0}: Error finding container 1fb8c86f100a10467d5f6faf7983505b8df8a0603e67fffc6ee732b41b33155b: Status 404 returned error can't find the container with id 1fb8c86f100a10467d5f6faf7983505b8df8a0603e67fffc6ee732b41b33155b
Sep 17 17:55:32.714214 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.714180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" event={"ID":"e51c716d-d95b-456e-859c-7f4babeeb36a","Type":"ContainerStarted","Data":"aa44cff5cd2dfb33230113f91880c8fba9b280ecf71b7d0da300075e84995f6c"}
Sep 17 17:55:32.714658 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.714239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" event={"ID":"e51c716d-d95b-456e-859c-7f4babeeb36a","Type":"ContainerStarted","Data":"1fb8c86f100a10467d5f6faf7983505b8df8a0603e67fffc6ee732b41b33155b"}
Sep 17 17:55:32.714658 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.714289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:55:32.715762 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.715730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" event={"ID":"ba108a53-9870-44a8-9aa9-b9a30e6c9851","Type":"ContainerStarted","Data":"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"}
Sep 17 17:55:32.715888 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.715780    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" event={"ID":"ba108a53-9870-44a8-9aa9-b9a30e6c9851","Type":"ContainerStarted","Data":"bf25fa86b79799b78c1ae23d08e165dee34f2023aef3a55b2c885a2be88337c8"}
Sep 17 17:55:32.715888 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.715825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:32.733138 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:32.733092    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" podStartSLOduration=1.733074175 podStartE2EDuration="1.733074175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:32.731496749 +0000 UTC m=+1292.591088103" watchObservedRunningTime="2026-09-17 17:55:32.733074175 +0000 UTC m=+1292.592665530"
Sep 17 17:55:33.722183 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:33.722148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" event={"ID":"ba108a53-9870-44a8-9aa9-b9a30e6c9851","Type":"ContainerStarted","Data":"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"}
Sep 17 17:55:33.754200 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:33.754149    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" podStartSLOduration=2.7541344949999997 podStartE2EDuration="2.754134495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:33.753010966 +0000 UTC m=+1293.612602322" watchObservedRunningTime="2026-09-17 17:55:33.754134495 +0000 UTC m=+1293.613725892"
Sep 17 17:55:33.974851 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:33.974778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:41.974211 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:41.974178    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:41.988040 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:41.988016    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:42.774583 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:42.774549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:55:42.776059 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:55:42.776033    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:56:03.723888 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:03.723861    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:56:06.905094 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:06.905057    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"]
Sep 17 17:56:06.906781 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:06.906744    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"]
Sep 17 17:56:06.915306 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:56:06.915266    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:56:06.915306 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.915306 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.915306 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a" cmd=["/bin/sleep","15"]
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:06.915335    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" message="" err=<
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:56:06.915349    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="main" containerID="cri-o://7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"
Sep 17 17:56:06.915593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:06.915391    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="main" containerID="cri-o://7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a" gracePeriod=60
Sep 17 17:56:07.132055 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.132022    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="llm-d-routing-sidecar" containerID="cri-o://3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43" gracePeriod=60
Sep 17 17:56:07.390954 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.390898    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:56:07.524128 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524040    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-model-cache\") pod \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") "
Sep 17 17:56:07.524128 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmfnn\" (UniqueName: \"kubernetes.io/projected/ba108a53-9870-44a8-9aa9-b9a30e6c9851-kube-api-access-xmfnn\") pod \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") "
Sep 17 17:56:07.524342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524143    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tls-certs\") pod \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") "
Sep 17 17:56:07.524342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524196    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-home\") pod \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") "
Sep 17 17:56:07.524342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tmp-dir\") pod \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") "
Sep 17 17:56:07.524342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-dshm\") pod \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\" (UID: \"ba108a53-9870-44a8-9aa9-b9a30e6c9851\") "
Sep 17 17:56:07.524342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba108a53-9870-44a8-9aa9-b9a30e6c9851" (UID: "ba108a53-9870-44a8-9aa9-b9a30e6c9851"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.524593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524484    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.524593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524524    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba108a53-9870-44a8-9aa9-b9a30e6c9851" (UID: "ba108a53-9870-44a8-9aa9-b9a30e6c9851"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.524593 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.524536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-home" (OuterVolumeSpecName: "home") pod "ba108a53-9870-44a8-9aa9-b9a30e6c9851" (UID: "ba108a53-9870-44a8-9aa9-b9a30e6c9851"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.526474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.526449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-dshm" (OuterVolumeSpecName: "dshm") pod "ba108a53-9870-44a8-9aa9-b9a30e6c9851" (UID: "ba108a53-9870-44a8-9aa9-b9a30e6c9851"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.526474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.526470    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba108a53-9870-44a8-9aa9-b9a30e6c9851" (UID: "ba108a53-9870-44a8-9aa9-b9a30e6c9851"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:07.526625 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.526558    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba108a53-9870-44a8-9aa9-b9a30e6c9851-kube-api-access-xmfnn" (OuterVolumeSpecName: "kube-api-access-xmfnn") pod "ba108a53-9870-44a8-9aa9-b9a30e6c9851" (UID: "ba108a53-9870-44a8-9aa9-b9a30e6c9851"). InnerVolumeSpecName "kube-api-access-xmfnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:07.625072 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.625040    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfnn\" (UniqueName: \"kubernetes.io/projected/ba108a53-9870-44a8-9aa9-b9a30e6c9851-kube-api-access-xmfnn\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.625072 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.625072    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.625271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.625086    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.625271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.625098    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.625271 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.625109    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba108a53-9870-44a8-9aa9-b9a30e6c9851-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.884130 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.884090    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerID="7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a" exitCode=0
Sep 17 17:56:07.884130 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.884119    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerID="3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43" exitCode=0
Sep 17 17:56:07.884486 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.884149    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"
Sep 17 17:56:07.884486 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.884177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" event={"ID":"ba108a53-9870-44a8-9aa9-b9a30e6c9851","Type":"ContainerDied","Data":"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"}
Sep 17 17:56:07.884486 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.884232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" event={"ID":"ba108a53-9870-44a8-9aa9-b9a30e6c9851","Type":"ContainerDied","Data":"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"}
Sep 17 17:56:07.884486 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.884251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd" event={"ID":"ba108a53-9870-44a8-9aa9-b9a30e6c9851","Type":"ContainerDied","Data":"bf25fa86b79799b78c1ae23d08e165dee34f2023aef3a55b2c885a2be88337c8"}
Sep 17 17:56:07.884486 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.884272    2566 scope.go:117] "RemoveContainer" containerID="7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"
Sep 17 17:56:07.893103 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.893085    2566 scope.go:117] "RemoveContainer" containerID="3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"
Sep 17 17:56:07.900447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.900428    2566 scope.go:117] "RemoveContainer" containerID="7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"
Sep 17 17:56:07.900716 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:56:07.900698    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a\": container with ID starting with 7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a not found: ID does not exist" containerID="7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"
Sep 17 17:56:07.900789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.900727    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"} err="failed to get container status \"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a\": rpc error: code = NotFound desc = could not find container \"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a\": container with ID starting with 7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a not found: ID does not exist"
Sep 17 17:56:07.900789 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.900752    2566 scope.go:117] "RemoveContainer" containerID="3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"
Sep 17 17:56:07.901020 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:56:07.901002    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43\": container with ID starting with 3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43 not found: ID does not exist" containerID="3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"
Sep 17 17:56:07.901065 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.901027    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"} err="failed to get container status \"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43\": rpc error: code = NotFound desc = could not find container \"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43\": container with ID starting with 3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43 not found: ID does not exist"
Sep 17 17:56:07.901065 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.901043    2566 scope.go:117] "RemoveContainer" containerID="7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"
Sep 17 17:56:07.901279 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.901257    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a"} err="failed to get container status \"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a\": rpc error: code = NotFound desc = could not find container \"7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a\": container with ID starting with 7385586928a826d7cd6da08610a35ae9d1ac9dc31c46ac91cf0e9bdc9b44d00a not found: ID does not exist"
Sep 17 17:56:07.901352 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.901281    2566 scope.go:117] "RemoveContainer" containerID="3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"
Sep 17 17:56:07.901511 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.901491    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43"} err="failed to get container status \"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43\": rpc error: code = NotFound desc = could not find container \"3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43\": container with ID starting with 3d9a548d949590a87a9af098b0ee2c5335129b5083edfdae40aedf168c702a43 not found: ID does not exist"
Sep 17 17:56:07.906971 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.906623    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"]
Sep 17 17:56:07.908388 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:07.908371    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qddrd"]
Sep 17 17:56:08.506488 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:08.506458    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" path="/var/lib/kubelet/pods/ba108a53-9870-44a8-9aa9-b9a30e6c9851/volumes"
Sep 17 17:56:13.724530 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:13.724447    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:21.925810 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:21.925771    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main" containerID="cri-o://aa44cff5cd2dfb33230113f91880c8fba9b280ecf71b7d0da300075e84995f6c" gracePeriod=45
Sep 17 17:56:23.724450 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:23.724413    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:33.724228 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:33.724189    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:33.724714 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:33.724294    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:56:43.724444 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:43.724404    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:52.066187 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.066154    2566 generic.go:356] "Generic (PLEG): container finished" podID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerID="aa44cff5cd2dfb33230113f91880c8fba9b280ecf71b7d0da300075e84995f6c" exitCode=0
Sep 17 17:56:52.066499 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.066227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" event={"ID":"e51c716d-d95b-456e-859c-7f4babeeb36a","Type":"ContainerDied","Data":"aa44cff5cd2dfb33230113f91880c8fba9b280ecf71b7d0da300075e84995f6c"}
Sep 17 17:56:52.174450 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.174425    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:56:52.314667 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.314638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc6jz\" (UniqueName: \"kubernetes.io/projected/e51c716d-d95b-456e-859c-7f4babeeb36a-kube-api-access-qc6jz\") pod \"e51c716d-d95b-456e-859c-7f4babeeb36a\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") "
Sep 17 17:56:52.314804 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.314687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c716d-d95b-456e-859c-7f4babeeb36a-tls-certs\") pod \"e51c716d-d95b-456e-859c-7f4babeeb36a\" (UID: \"e51c716d-d95b-456e-859c-7f4babeeb36a\") "
Sep 17 17:56:52.316640 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.316613    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51c716d-d95b-456e-859c-7f4babeeb36a-kube-api-access-qc6jz" (OuterVolumeSpecName: "kube-api-access-qc6jz") pod "e51c716d-d95b-456e-859c-7f4babeeb36a" (UID: "e51c716d-d95b-456e-859c-7f4babeeb36a"). InnerVolumeSpecName "kube-api-access-qc6jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:52.316744 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.316686    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51c716d-d95b-456e-859c-7f4babeeb36a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e51c716d-d95b-456e-859c-7f4babeeb36a" (UID: "e51c716d-d95b-456e-859c-7f4babeeb36a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:52.415462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.415427    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6jz\" (UniqueName: \"kubernetes.io/projected/e51c716d-d95b-456e-859c-7f4babeeb36a-kube-api-access-qc6jz\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:52.415462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:52.415457    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c716d-d95b-456e-859c-7f4babeeb36a-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:56:53.071766 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:53.071730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626" event={"ID":"e51c716d-d95b-456e-859c-7f4babeeb36a","Type":"ContainerDied","Data":"1fb8c86f100a10467d5f6faf7983505b8df8a0603e67fffc6ee732b41b33155b"}
Sep 17 17:56:53.072175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:53.071779    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"
Sep 17 17:56:53.072175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:53.071819    2566 scope.go:117] "RemoveContainer" containerID="aa44cff5cd2dfb33230113f91880c8fba9b280ecf71b7d0da300075e84995f6c"
Sep 17 17:56:53.095594 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:53.095570    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"]
Sep 17 17:56:53.101368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:53.101347    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsf626"]
Sep 17 17:56:54.506397 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:56:54.506352    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" path="/var/lib/kubelet/pods/e51c716d-d95b-456e-859c-7f4babeeb36a/volumes"
Sep 17 17:57:19.742676 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.742641    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"]
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.742986    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="llm-d-routing-sidecar"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743000    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="llm-d-routing-sidecar"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743024    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743031    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743043    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="main"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743049    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="main"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743103    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="llm-d-routing-sidecar"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743112    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51c716d-d95b-456e-859c-7f4babeeb36a" containerName="main"
Sep 17 17:57:19.743146 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.743118    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba108a53-9870-44a8-9aa9-b9a30e6c9851" containerName="main"
Sep 17 17:57:19.747754 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.747733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.750434 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.750411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 17:57:19.751587 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.751564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:19.751708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.751570    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 17:57:19.751708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.751621    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 17:57:19.751708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.751635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 17:57:19.751708 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.751648    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-86qsh\""
Sep 17 17:57:19.758859 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.758838    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"]
Sep 17 17:57:19.827341 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-home\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.827487 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mgwv\" (UniqueName: \"kubernetes.io/projected/89af777c-61be-4539-84ff-239996fa7ae3-kube-api-access-8mgwv\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.827487 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.827487 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-model-cache\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.827487 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.827667 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89af777c-61be-4539-84ff-239996fa7ae3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.827667 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-dshm\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.827667 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.827544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89af777c-61be-4539-84ff-239996fa7ae3-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928422 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-model-cache\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89af777c-61be-4539-84ff-239996fa7ae3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-dshm\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89af777c-61be-4539-84ff-239996fa7ae3-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928563 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-home\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928841 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mgwv\" (UniqueName: \"kubernetes.io/projected/89af777c-61be-4539-84ff-239996fa7ae3-kube-api-access-8mgwv\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928841 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928998 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-model-cache\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.928998 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.928956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.929167 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.929146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-home\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.929231 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.929155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.929444 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.929421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89af777c-61be-4539-84ff-239996fa7ae3-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.930826 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.930804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-dshm\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.931012 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.930997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89af777c-61be-4539-84ff-239996fa7ae3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:19.938310 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:19.938286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mgwv\" (UniqueName: \"kubernetes.io/projected/89af777c-61be-4539-84ff-239996fa7ae3-kube-api-access-8mgwv\") pod \"tokenizer-clean-path-test-kserve-7948d59869-9x2nn\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:20.001386 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.001321    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"]
Sep 17 17:57:20.005881 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.005861    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.020368 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.020338    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"]
Sep 17 17:57:20.058466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.058439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:20.131086 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.131038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.131226 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.131093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.131226 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.131154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.131226 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.131209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.231845 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.231810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.232014 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.231885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.232014 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.231906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.232014 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.231936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.232285 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.232262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.232328 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.232272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.232576 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.232559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.234385 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.234362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.318244 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.318223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:57:20.339794 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.339767    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"]
Sep 17 17:57:20.342356 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:57:20.342326    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89af777c_61be_4539_84ff_239996fa7ae3.slice/crio-c02d06c5b69463f58f40d4cd02a735c595c5c74b5d36f22977b2de4ac5c1be89 WatchSource:0}: Error finding container c02d06c5b69463f58f40d4cd02a735c595c5c74b5d36f22977b2de4ac5c1be89: Status 404 returned error can't find the container with id c02d06c5b69463f58f40d4cd02a735c595c5c74b5d36f22977b2de4ac5c1be89
Sep 17 17:57:20.592245 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:20.592147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"]
Sep 17 17:57:20.594943 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:57:20.594898    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584b0d5e_5819_4249_8c7d_cc7f3e54d1b4.slice/crio-ee32664df6e49fb67e6b046074108d3c3cb604d1798a077069b66eb0e8dbcac9 WatchSource:0}: Error finding container ee32664df6e49fb67e6b046074108d3c3cb604d1798a077069b66eb0e8dbcac9: Status 404 returned error can't find the container with id ee32664df6e49fb67e6b046074108d3c3cb604d1798a077069b66eb0e8dbcac9
Sep 17 17:57:21.186568 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:21.186525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" event={"ID":"89af777c-61be-4539-84ff-239996fa7ae3","Type":"ContainerStarted","Data":"955397ffe202c473480909f3aeb909d52118a6370164fbc2e3c9850d1f1cbe73"}
Sep 17 17:57:21.187041 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:21.186602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" event={"ID":"89af777c-61be-4539-84ff-239996fa7ae3","Type":"ContainerStarted","Data":"c02d06c5b69463f58f40d4cd02a735c595c5c74b5d36f22977b2de4ac5c1be89"}
Sep 17 17:57:21.188282 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:21.188258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" event={"ID":"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4","Type":"ContainerStarted","Data":"14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5"}
Sep 17 17:57:21.188389 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:21.188304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" event={"ID":"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4","Type":"ContainerStarted","Data":"ee32664df6e49fb67e6b046074108d3c3cb604d1798a077069b66eb0e8dbcac9"}
Sep 17 17:57:23.199960 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:23.199899    2566 generic.go:356] "Generic (PLEG): container finished" podID="89af777c-61be-4539-84ff-239996fa7ae3" containerID="955397ffe202c473480909f3aeb909d52118a6370164fbc2e3c9850d1f1cbe73" exitCode=0
Sep 17 17:57:23.199960 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:23.199948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" event={"ID":"89af777c-61be-4539-84ff-239996fa7ae3","Type":"ContainerDied","Data":"955397ffe202c473480909f3aeb909d52118a6370164fbc2e3c9850d1f1cbe73"}
Sep 17 17:57:23.201575 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:23.201552    2566 generic.go:356] "Generic (PLEG): container finished" podID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerID="14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5" exitCode=0
Sep 17 17:57:23.201653 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:23.201629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" event={"ID":"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4","Type":"ContainerDied","Data":"14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5"}
Sep 17 17:57:23.202725 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:23.202711    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:57:24.211341 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:24.211299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" event={"ID":"89af777c-61be-4539-84ff-239996fa7ae3","Type":"ContainerStarted","Data":"b61e21ca70ab468bf94da26ea3a97013844405dbb24ff5913ff873930993fa17"}
Sep 17 17:57:24.234609 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:24.234561    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" podStartSLOduration=2.763236033 podStartE2EDuration="5.234540561s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.471304528s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:24.232626433 +0000 UTC m=+1404.092217789" watchObservedRunningTime="2026-09-17 17:57:24.234540561 +0000 UTC m=+1404.094131920"
Sep 17 17:57:25.059470 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:25.059435    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:30.059717 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:30.059670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:30.078281 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:30.078253    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:57:30.261643 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:57:30.261575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:58:09.447633 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:09.447580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" event={"ID":"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4","Type":"ContainerStarted","Data":"0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98"}
Sep 17 17:58:09.448650 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:09.448622    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:58:09.471096 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:09.470412    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" podStartSLOduration=2.190998913 podStartE2EDuration="50.470376674s" totalImagesPullingTime="45.759717145s" totalInitContainerRuntime="2.519660616s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:09.470023892 +0000 UTC m=+1449.329615248" watchObservedRunningTime="2026-09-17 17:58:09.470376674 +0000 UTC m=+1449.329968031"
Sep 17 17:58:40.463632 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:40.463595    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:58:41.768692 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:41.768663    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8_584b0d5e-5819-4249-8c7d-cc7f3e54d1b4/main/0.log"
Sep 17 17:58:41.957555 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:41.957520    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"]
Sep 17 17:58:41.957844 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:41.957811    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" podUID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerName="main" containerID="cri-o://0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98" gracePeriod=30
Sep 17 17:58:41.962022 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:41.961994    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"]
Sep 17 17:58:41.971334 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:58:41.971038    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:58:41.971334 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:41.971334 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:41.971334 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="b61e21ca70ab468bf94da26ea3a97013844405dbb24ff5913ff873930993fa17" cmd=["/bin/sleep","15"]
Sep 17 17:58:41.971693 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:41.971669    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" message="" err=<
Sep 17 17:58:41.971693 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:41.971693 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:41.971693 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 17:58:41.971951 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:58:41.971931    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:58:41.971951 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:41.971951 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:41.971951 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="main" containerID="cri-o://b61e21ca70ab468bf94da26ea3a97013844405dbb24ff5913ff873930993fa17"
Sep 17 17:58:41.972238 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:41.972206    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="main" containerID="cri-o://b61e21ca70ab468bf94da26ea3a97013844405dbb24ff5913ff873930993fa17" gracePeriod=60
Sep 17 17:58:42.245568 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.245513    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 17:58:42.573658 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.573632    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:58:42.594660 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.594593    2566 generic.go:356] "Generic (PLEG): container finished" podID="89af777c-61be-4539-84ff-239996fa7ae3" containerID="b61e21ca70ab468bf94da26ea3a97013844405dbb24ff5913ff873930993fa17" exitCode=0
Sep 17 17:58:42.594801 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.594664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" event={"ID":"89af777c-61be-4539-84ff-239996fa7ae3","Type":"ContainerDied","Data":"b61e21ca70ab468bf94da26ea3a97013844405dbb24ff5913ff873930993fa17"}
Sep 17 17:58:42.596437 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.596416    2566 generic.go:356] "Generic (PLEG): container finished" podID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerID="0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98" exitCode=0
Sep 17 17:58:42.596556 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.596445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" event={"ID":"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4","Type":"ContainerDied","Data":"0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98"}
Sep 17 17:58:42.596556 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.596466    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"
Sep 17 17:58:42.596556 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.596475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8" event={"ID":"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4","Type":"ContainerDied","Data":"ee32664df6e49fb67e6b046074108d3c3cb604d1798a077069b66eb0e8dbcac9"}
Sep 17 17:58:42.596556 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.596504    2566 scope.go:117] "RemoveContainer" containerID="0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98"
Sep 17 17:58:42.604927 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.604892    2566 scope.go:117] "RemoveContainer" containerID="14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5"
Sep 17 17:58:42.612646 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.612627    2566 scope.go:117] "RemoveContainer" containerID="0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98"
Sep 17 17:58:42.612938 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:58:42.612920    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98\": container with ID starting with 0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98 not found: ID does not exist" containerID="0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98"
Sep 17 17:58:42.613036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.612944    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98"} err="failed to get container status \"0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98\": rpc error: code = NotFound desc = could not find container \"0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98\": container with ID starting with 0c09d141fbf19aba5986b2752b1933e921874c0e0c1d5b0026c643da547a8b98 not found: ID does not exist"
Sep 17 17:58:42.613036 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.612961    2566 scope.go:117] "RemoveContainer" containerID="14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5"
Sep 17 17:58:42.613222 ip-10-0-135-84 kubenswrapper[2566]: E0917 17:58:42.613198    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5\": container with ID starting with 14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5 not found: ID does not exist" containerID="14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5"
Sep 17 17:58:42.613294 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.613228    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5"} err="failed to get container status \"14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5\": rpc error: code = NotFound desc = could not find container \"14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5\": container with ID starting with 14db3bc5e739b0e2ce9a0c8ae7c71d7bd9f62849f9582b555afeb5b810105ee5 not found: ID does not exist"
Sep 17 17:58:42.701812 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.701785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-kserve-provision-location\") pod \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") "
Sep 17 17:58:42.701981 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.701823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tls-certs\") pod \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") "
Sep 17 17:58:42.701981 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.701886    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tmp-dir\") pod \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") "
Sep 17 17:58:42.701981 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.701941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-cabundle-cert\") pod \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\" (UID: \"584b0d5e-5819-4249-8c7d-cc7f3e54d1b4\") "
Sep 17 17:58:42.702340 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.702314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" (UID: "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.702402 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.702322    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" (UID: "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:42.703942 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.703924    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" (UID: "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:42.723030 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.723000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" (UID: "584b0d5e-5819-4249-8c7d-cc7f3e54d1b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.803263 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.803236    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.803263 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.803261    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.803629 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.803273    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.803629 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.803281    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.860203 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.860139    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:58:42.904355 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-tmp-dir\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904533 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904403    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-kserve-provision-location\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904533 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-model-cache\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904533 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904461    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89af777c-61be-4539-84ff-239996fa7ae3-cabundle-cert\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904533 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904494    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mgwv\" (UniqueName: \"kubernetes.io/projected/89af777c-61be-4539-84ff-239996fa7ae3-kube-api-access-8mgwv\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904533 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-home\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904543    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89af777c-61be-4539-84ff-239996fa7ae3-tls-certs\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904583    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-dshm\") pod \"89af777c-61be-4539-84ff-239996fa7ae3\" (UID: \"89af777c-61be-4539-84ff-239996fa7ae3\") "
Sep 17 17:58:42.904838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904609    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.904838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904699    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-model-cache" (OuterVolumeSpecName: "model-cache") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.904838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89af777c-61be-4539-84ff-239996fa7ae3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:42.904838 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904815    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-home" (OuterVolumeSpecName: "home") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.905321 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904882    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.905321 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904901    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.905321 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904943    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.905321 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.904959    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89af777c-61be-4539-84ff-239996fa7ae3-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.907252 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.907222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-dshm" (OuterVolumeSpecName: "dshm") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.907388 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.907347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89af777c-61be-4539-84ff-239996fa7ae3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:42.907388 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.907358    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89af777c-61be-4539-84ff-239996fa7ae3-kube-api-access-8mgwv" (OuterVolumeSpecName: "kube-api-access-8mgwv") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "kube-api-access-8mgwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:42.921086 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.921065    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"]
Sep 17 17:58:42.925245 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.925211    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89af777c-61be-4539-84ff-239996fa7ae3" (UID: "89af777c-61be-4539-84ff-239996fa7ae3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.926452 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:42.926433    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cd6fcf768-cc4z8"]
Sep 17 17:58:43.006173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.006146    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mgwv\" (UniqueName: \"kubernetes.io/projected/89af777c-61be-4539-84ff-239996fa7ae3-kube-api-access-8mgwv\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:43.006173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.006169    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89af777c-61be-4539-84ff-239996fa7ae3-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:43.006173 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.006180    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:43.006363 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.006188    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89af777c-61be-4539-84ff-239996fa7ae3-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 17:58:43.602259 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.602223    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"
Sep 17 17:58:43.602259 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.602237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn" event={"ID":"89af777c-61be-4539-84ff-239996fa7ae3","Type":"ContainerDied","Data":"c02d06c5b69463f58f40d4cd02a735c595c5c74b5d36f22977b2de4ac5c1be89"}
Sep 17 17:58:43.602514 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.602294    2566 scope.go:117] "RemoveContainer" containerID="b61e21ca70ab468bf94da26ea3a97013844405dbb24ff5913ff873930993fa17"
Sep 17 17:58:43.611658 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.611643    2566 scope.go:117] "RemoveContainer" containerID="955397ffe202c473480909f3aeb909d52118a6370164fbc2e3c9850d1f1cbe73"
Sep 17 17:58:43.626398 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.626376    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"]
Sep 17 17:58:43.631319 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:43.631300    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-9x2nn"]
Sep 17 17:58:44.506547 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:44.506516    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" path="/var/lib/kubelet/pods/584b0d5e-5819-4249-8c7d-cc7f3e54d1b4/volumes"
Sep 17 17:58:44.506983 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:58:44.506969    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89af777c-61be-4539-84ff-239996fa7ae3" path="/var/lib/kubelet/pods/89af777c-61be-4539-84ff-239996fa7ae3/volumes"
Sep 17 17:59:02.491357 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:02.491334    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:59:02.493634 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:02.493615    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 17:59:57.421130 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421091    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"]
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421398    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="storage-initializer"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421408    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="storage-initializer"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421418    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="main"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421423    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="main"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421433    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerName="main"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421439    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerName="main"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421450    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerName="storage-initializer"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421455    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerName="storage-initializer"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421508    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="584b0d5e-5819-4249-8c7d-cc7f3e54d1b4" containerName="main"
Sep 17 17:59:57.422474 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.421519    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="89af777c-61be-4539-84ff-239996fa7ae3" containerName="main"
Sep 17 17:59:57.423342 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.423326    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.425879 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.425856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 17:59:57.426329 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.426311    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 17:59:57.427032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.427016    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 17:59:57.427032 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.427025    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 17:59:57.427163 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.427027    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:59:57.427326 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.427313    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-qbq5r\""
Sep 17 17:59:57.440018 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.439997    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"]
Sep 17 17:59:57.527023 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.526986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-model-cache\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.527175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.527036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-dshm\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.527175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.527095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tls-certs\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.527175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.527132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-home\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.527175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.527150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.527175 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.527168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.527346 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.527186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.527346 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.527242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vtj\" (UniqueName: \"kubernetes.io/projected/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kube-api-access-c8vtj\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628462 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-home\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628633 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628633 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628633 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628633 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vtj\" (UniqueName: \"kubernetes.io/projected/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kube-api-access-c8vtj\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628851 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-model-cache\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628851 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-dshm\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.628851 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tls-certs\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.629035 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.629035 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.628887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-home\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.629466 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.629408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.629608 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.629585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.629763 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.629739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-model-cache\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.631400 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.631381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tls-certs\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.631535 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.631514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-dshm\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.639022 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.639002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vtj\" (UniqueName: \"kubernetes.io/projected/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kube-api-access-c8vtj\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.660715 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.660691    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"]
Sep 17 17:59:57.664095 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.664076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.673821 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.673791    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"]
Sep 17 17:59:57.734060 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.734034    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 17:59:57.830158 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.830116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422fed61-437f-4989-a941-3ccdc0ad2791-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.830325 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.830180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.830325 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.830221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.830325 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.830268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/422fed61-437f-4989-a941-3ccdc0ad2791-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.931139 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.931065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422fed61-437f-4989-a941-3ccdc0ad2791-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.931139 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.931115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.931139 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.931139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.931362 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.931184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/422fed61-437f-4989-a941-3ccdc0ad2791-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.931578 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.931558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.931657 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.931619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.931924 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.931892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/422fed61-437f-4989-a941-3ccdc0ad2791-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.933670 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.933645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422fed61-437f-4989-a941-3ccdc0ad2791-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:57.977217 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:57.977190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 17:59:58.006123 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:58.006096    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"]
Sep 17 17:59:58.007983 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:59:58.007955    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af4fd28_f60f_4b2a_8ebe_c72709fc7fee.slice/crio-58579bf99be441654e80d5ccc70e8c46c7eff77e69920a7a196954c1a28462c1 WatchSource:0}: Error finding container 58579bf99be441654e80d5ccc70e8c46c7eff77e69920a7a196954c1a28462c1: Status 404 returned error can't find the container with id 58579bf99be441654e80d5ccc70e8c46c7eff77e69920a7a196954c1a28462c1
Sep 17 17:59:58.250447 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:58.250419    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"]
Sep 17 17:59:58.252349 ip-10-0-135-84 kubenswrapper[2566]: W0917 17:59:58.252309    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422fed61_437f_4989_a941_3ccdc0ad2791.slice/crio-1f622b47feb5d1784af44ededeb1578762d64960c8172be54dc8b30cfe470f28 WatchSource:0}: Error finding container 1f622b47feb5d1784af44ededeb1578762d64960c8172be54dc8b30cfe470f28: Status 404 returned error can't find the container with id 1f622b47feb5d1784af44ededeb1578762d64960c8172be54dc8b30cfe470f28
Sep 17 17:59:58.920152 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:58.920113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" event={"ID":"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee","Type":"ContainerStarted","Data":"e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46"}
Sep 17 17:59:58.920630 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:58.920176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" event={"ID":"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee","Type":"ContainerStarted","Data":"58579bf99be441654e80d5ccc70e8c46c7eff77e69920a7a196954c1a28462c1"}
Sep 17 17:59:58.921766 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:58.921737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" event={"ID":"422fed61-437f-4989-a941-3ccdc0ad2791","Type":"ContainerStarted","Data":"2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67"}
Sep 17 17:59:58.921864 ip-10-0-135-84 kubenswrapper[2566]: I0917 17:59:58.921798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" event={"ID":"422fed61-437f-4989-a941-3ccdc0ad2791","Type":"ContainerStarted","Data":"1f622b47feb5d1784af44ededeb1578762d64960c8172be54dc8b30cfe470f28"}
Sep 17 18:00:00.932079 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:00.932047    2566 generic.go:356] "Generic (PLEG): container finished" podID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerID="e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46" exitCode=0
Sep 17 18:00:00.932461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:00.932119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" event={"ID":"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee","Type":"ContainerDied","Data":"e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46"}
Sep 17 18:00:00.933741 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:00.933720    2566 generic.go:356] "Generic (PLEG): container finished" podID="422fed61-437f-4989-a941-3ccdc0ad2791" containerID="2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67" exitCode=0
Sep 17 18:00:00.933843 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:00.933779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" event={"ID":"422fed61-437f-4989-a941-3ccdc0ad2791","Type":"ContainerDied","Data":"2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67"}
Sep 17 18:00:01.940336 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:01.940298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" event={"ID":"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee","Type":"ContainerStarted","Data":"b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c"}
Sep 17 18:00:01.942485 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:01.942457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" event={"ID":"422fed61-437f-4989-a941-3ccdc0ad2791","Type":"ContainerStarted","Data":"aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c"}
Sep 17 18:00:01.942688 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:01.942673    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 18:00:01.964435 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:01.964385    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" podStartSLOduration=2.423175389 podStartE2EDuration="4.964366151s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.541190762s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:01.963357233 +0000 UTC m=+1561.822948589" watchObservedRunningTime="2026-09-17 18:00:01.964366151 +0000 UTC m=+1561.823957530"
Sep 17 18:00:01.987989 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:01.987943    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" podStartSLOduration=2.409303162 podStartE2EDuration="4.987903836s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.578600674s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:01.987203979 +0000 UTC m=+1561.846795336" watchObservedRunningTime="2026-09-17 18:00:01.987903836 +0000 UTC m=+1561.847495192"
Sep 17 18:00:02.734138 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:02.734098    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 18:00:07.734745 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:07.734716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 18:00:07.747744 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:07.747718    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 18:00:07.980824 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:07.980797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 18:00:32.959603 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:32.959568    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 18:00:46.234661 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:46.234558    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc_422fed61-437f-4989-a941-3ccdc0ad2791/main/0.log"
Sep 17 18:00:48.202774 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.202734    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"]
Sep 17 18:00:48.211796 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:00:48.211754    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:00:48.211796 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:48.211796 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:48.211796 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c" cmd=["/bin/sleep","15"]
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.211814    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" message="" err=<
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:00:48.211830    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerName="main" containerID="cri-o://b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c"
Sep 17 18:00:48.212196 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.211883    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerName="main" containerID="cri-o://b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c" gracePeriod=60
Sep 17 18:00:48.214722 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.214701    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"]
Sep 17 18:00:48.215016 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.214968    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" podUID="422fed61-437f-4989-a941-3ccdc0ad2791" containerName="main" containerID="cri-o://aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c" gracePeriod=30
Sep 17 18:00:48.561172 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.561144    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 18:00:48.704371 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704337    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-dshm\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704386    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tmp-dir\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8vtj\" (UniqueName: \"kubernetes.io/projected/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kube-api-access-c8vtj\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-home\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704684 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-model-cache\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704684 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704579    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kserve-provision-location\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704684 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704612    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tls-certs\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704684 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-cabundle-cert\") pod \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\" (UID: \"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee\") "
Sep 17 18:00:48.704868 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704698    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.704945 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-model-cache" (OuterVolumeSpecName: "model-cache") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.705010 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.704982    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.705073 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.705033    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.705231 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.705206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:48.705502 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.705467    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-home" (OuterVolumeSpecName: "home") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.708077 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.708052    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-dshm" (OuterVolumeSpecName: "dshm") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.708191 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.708135    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kube-api-access-c8vtj" (OuterVolumeSpecName: "kube-api-access-c8vtj") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "kube-api-access-c8vtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:48.708191 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.708138    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:48.728896 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.728834    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" (UID: "0af4fd28-f60f-4b2a-8ebe-c72709fc7fee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.773121 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.773099    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 18:00:48.806233 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.806205    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.806352 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.806238    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.806352 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.806255    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.806352 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.806269    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.806352 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.806282    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.806352 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.806294    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vtj\" (UniqueName: \"kubernetes.io/projected/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee-kube-api-access-c8vtj\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.907551 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.907518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-kserve-provision-location\") pod \"422fed61-437f-4989-a941-3ccdc0ad2791\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") "
Sep 17 18:00:48.907703 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.907598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422fed61-437f-4989-a941-3ccdc0ad2791-tls-certs\") pod \"422fed61-437f-4989-a941-3ccdc0ad2791\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") "
Sep 17 18:00:48.907703 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.907627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/422fed61-437f-4989-a941-3ccdc0ad2791-cabundle-cert\") pod \"422fed61-437f-4989-a941-3ccdc0ad2791\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") "
Sep 17 18:00:48.907703 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.907688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-tmp-dir\") pod \"422fed61-437f-4989-a941-3ccdc0ad2791\" (UID: \"422fed61-437f-4989-a941-3ccdc0ad2791\") "
Sep 17 18:00:48.908073 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.908039    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422fed61-437f-4989-a941-3ccdc0ad2791-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "422fed61-437f-4989-a941-3ccdc0ad2791" (UID: "422fed61-437f-4989-a941-3ccdc0ad2791"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:48.908186 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.908138    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "422fed61-437f-4989-a941-3ccdc0ad2791" (UID: "422fed61-437f-4989-a941-3ccdc0ad2791"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.909764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.909744    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422fed61-437f-4989-a941-3ccdc0ad2791-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "422fed61-437f-4989-a941-3ccdc0ad2791" (UID: "422fed61-437f-4989-a941-3ccdc0ad2791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:48.930363 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:48.930327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "422fed61-437f-4989-a941-3ccdc0ad2791" (UID: "422fed61-437f-4989-a941-3ccdc0ad2791"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:49.008465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.008391    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:49.008465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.008417    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/422fed61-437f-4989-a941-3ccdc0ad2791-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:49.008465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.008429    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422fed61-437f-4989-a941-3ccdc0ad2791-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:49.008465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.008443    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/422fed61-437f-4989-a941-3ccdc0ad2791-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:00:49.148122 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.148081    2566 generic.go:356] "Generic (PLEG): container finished" podID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerID="b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c" exitCode=0
Sep 17 18:00:49.148303 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.148131    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"
Sep 17 18:00:49.148303 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.148172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" event={"ID":"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee","Type":"ContainerDied","Data":"b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c"}
Sep 17 18:00:49.148303 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.148229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh" event={"ID":"0af4fd28-f60f-4b2a-8ebe-c72709fc7fee","Type":"ContainerDied","Data":"58579bf99be441654e80d5ccc70e8c46c7eff77e69920a7a196954c1a28462c1"}
Sep 17 18:00:49.148303 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.148252    2566 scope.go:117] "RemoveContainer" containerID="b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c"
Sep 17 18:00:49.149965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.149938    2566 generic.go:356] "Generic (PLEG): container finished" podID="422fed61-437f-4989-a941-3ccdc0ad2791" containerID="aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c" exitCode=0
Sep 17 18:00:49.150103 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.149983    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"
Sep 17 18:00:49.150103 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.149997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" event={"ID":"422fed61-437f-4989-a941-3ccdc0ad2791","Type":"ContainerDied","Data":"aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c"}
Sep 17 18:00:49.150103 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.150040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc" event={"ID":"422fed61-437f-4989-a941-3ccdc0ad2791","Type":"ContainerDied","Data":"1f622b47feb5d1784af44ededeb1578762d64960c8172be54dc8b30cfe470f28"}
Sep 17 18:00:49.157395 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.157255    2566 scope.go:117] "RemoveContainer" containerID="e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46"
Sep 17 18:00:49.164889 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.164872    2566 scope.go:117] "RemoveContainer" containerID="b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c"
Sep 17 18:00:49.165200 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:00:49.165183    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c\": container with ID starting with b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c not found: ID does not exist" containerID="b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c"
Sep 17 18:00:49.165258 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.165208    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c"} err="failed to get container status \"b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c\": rpc error: code = NotFound desc = could not find container \"b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c\": container with ID starting with b964b79cdaebb2d2f7d31baa887ab54f3710286303ec65e624494c719bd79d8c not found: ID does not exist"
Sep 17 18:00:49.165258 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.165224    2566 scope.go:117] "RemoveContainer" containerID="e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46"
Sep 17 18:00:49.165446 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:00:49.165426    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46\": container with ID starting with e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46 not found: ID does not exist" containerID="e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46"
Sep 17 18:00:49.165509 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.165456    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46"} err="failed to get container status \"e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46\": rpc error: code = NotFound desc = could not find container \"e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46\": container with ID starting with e23431a399e8beec9556280c0539a6d05f1c143067772588af01a0b2de33ce46 not found: ID does not exist"
Sep 17 18:00:49.165509 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.165479    2566 scope.go:117] "RemoveContainer" containerID="aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c"
Sep 17 18:00:49.172560 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.172547    2566 scope.go:117] "RemoveContainer" containerID="2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67"
Sep 17 18:00:49.179476 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.179451    2566 scope.go:117] "RemoveContainer" containerID="aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c"
Sep 17 18:00:49.179707 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:00:49.179690    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c\": container with ID starting with aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c not found: ID does not exist" containerID="aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c"
Sep 17 18:00:49.179777 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.179716    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c"} err="failed to get container status \"aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c\": rpc error: code = NotFound desc = could not find container \"aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c\": container with ID starting with aeb9fa5f194a5083bcb0840c3fe0bddad4a6a36a43937b1d89bed1fede946b2c not found: ID does not exist"
Sep 17 18:00:49.179777 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.179748    2566 scope.go:117] "RemoveContainer" containerID="2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67"
Sep 17 18:00:49.179990 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:00:49.179973    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67\": container with ID starting with 2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67 not found: ID does not exist" containerID="2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67"
Sep 17 18:00:49.180046 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.179993    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67"} err="failed to get container status \"2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67\": rpc error: code = NotFound desc = could not find container \"2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67\": container with ID starting with 2d891f28bcffe11f6fb295f0bbabf024fce575d2be55289c334237d3c51c9c67 not found: ID does not exist"
Sep 17 18:00:49.182807 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.182787    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"]
Sep 17 18:00:49.193047 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.193026    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-ht4kh"]
Sep 17 18:00:49.206198 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.206173    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"]
Sep 17 18:00:49.221886 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:49.221862    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-785b85b594-rrtcc"]
Sep 17 18:00:50.506682 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:50.506641    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" path="/var/lib/kubelet/pods/0af4fd28-f60f-4b2a-8ebe-c72709fc7fee/volumes"
Sep 17 18:00:50.507267 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:00:50.507247    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="422fed61-437f-4989-a941-3ccdc0ad2791" path="/var/lib/kubelet/pods/422fed61-437f-4989-a941-3ccdc0ad2791/volumes"
Sep 17 18:01:55.425106 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425066    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"]
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425529    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerName="main"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425548    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerName="main"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425569    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerName="storage-initializer"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425577    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerName="storage-initializer"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425589    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422fed61-437f-4989-a941-3ccdc0ad2791" containerName="storage-initializer"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425598    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="422fed61-437f-4989-a941-3ccdc0ad2791" containerName="storage-initializer"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425613    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422fed61-437f-4989-a941-3ccdc0ad2791" containerName="main"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425621    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="422fed61-437f-4989-a941-3ccdc0ad2791" containerName="main"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425720    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="422fed61-437f-4989-a941-3ccdc0ad2791" containerName="main"
Sep 17 18:01:55.425746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.425734    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af4fd28-f60f-4b2a-8ebe-c72709fc7fee" containerName="main"
Sep 17 18:01:55.427951 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.427931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.430562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.430541    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:55.430665 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.430546    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-twtxs\""
Sep 17 18:01:55.430737 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.430720    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:55.430851 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.430824    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:01:55.431808 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.431788    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:01:55.431901 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.431853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:01:55.441864 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.441837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"]
Sep 17 18:01:55.469544 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-tmp-dir\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.469544 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xct72\" (UniqueName: \"kubernetes.io/projected/e9a7eb37-3927-420a-85ce-48eab3feba64-kube-api-access-xct72\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.469798 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7eb37-3927-420a-85ce-48eab3feba64-tls-certs\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.469798 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-dshm\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.469798 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.469798 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-model-cache\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.469798 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9a7eb37-3927-420a-85ce-48eab3feba64-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.470103 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.469829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-home\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570235 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-dshm\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570446 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570446 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-model-cache\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9a7eb37-3927-420a-85ce-48eab3feba64-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-home\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570711 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-tmp-dir\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570770 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570770 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct72\" (UniqueName: \"kubernetes.io/projected/e9a7eb37-3927-420a-85ce-48eab3feba64-kube-api-access-xct72\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570770 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-model-cache\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.570947 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7eb37-3927-420a-85ce-48eab3feba64-tls-certs\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.571009 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-home\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.571009 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.570993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-tmp-dir\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.571217 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.571196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9a7eb37-3927-420a-85ce-48eab3feba64-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.572655 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.572634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-dshm\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.573126 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.573108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7eb37-3927-420a-85ce-48eab3feba64-tls-certs\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.579306 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.579281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct72\" (UniqueName: \"kubernetes.io/projected/e9a7eb37-3927-420a-85ce-48eab3feba64-kube-api-access-xct72\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-mx6v6\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.738608 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.738535    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:01:55.771460 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.771016    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"]
Sep 17 18:01:55.774519 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.774496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.787360 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.787335    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"]
Sep 17 18:01:55.873713 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.873683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a930b54c-e0da-4531-8da3-739d44c467a9-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.873713 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.873718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.873937 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.873737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.873937 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.873818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a930b54c-e0da-4531-8da3-739d44c467a9-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.975259 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.975223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a930b54c-e0da-4531-8da3-739d44c467a9-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.975423 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.975266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.975423 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.975284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.975423 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.975401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a930b54c-e0da-4531-8da3-739d44c467a9-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.975627 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.975610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.975704 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.975685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.976081 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.976058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a930b54c-e0da-4531-8da3-739d44c467a9-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:55.977813 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:55.977796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a930b54c-e0da-4531-8da3-739d44c467a9-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69854679d-bqh9g\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:56.017600 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:56.017579    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"]
Sep 17 18:01:56.019421 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:01:56.019401    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a7eb37_3927_420a_85ce_48eab3feba64.slice/crio-eb99a175ca6db3b3bbf69e3e77a59b923b3b99e2a6d831524f777aab4e65da62 WatchSource:0}: Error finding container eb99a175ca6db3b3bbf69e3e77a59b923b3b99e2a6d831524f777aab4e65da62: Status 404 returned error can't find the container with id eb99a175ca6db3b3bbf69e3e77a59b923b3b99e2a6d831524f777aab4e65da62
Sep 17 18:01:56.089981 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:56.089960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:01:56.359632 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:56.359601    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"]
Sep 17 18:01:56.362935 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:01:56.362890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda930b54c_e0da_4531_8da3_739d44c467a9.slice/crio-ba02fce5dc14de46010577ca3635b6c9eb59226f13deeccb2f4342e54110d68f WatchSource:0}: Error finding container ba02fce5dc14de46010577ca3635b6c9eb59226f13deeccb2f4342e54110d68f: Status 404 returned error can't find the container with id ba02fce5dc14de46010577ca3635b6c9eb59226f13deeccb2f4342e54110d68f
Sep 17 18:01:56.431294 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:56.431260    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" event={"ID":"e9a7eb37-3927-420a-85ce-48eab3feba64","Type":"ContainerStarted","Data":"ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a"}
Sep 17 18:01:56.431685 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:56.431327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" event={"ID":"e9a7eb37-3927-420a-85ce-48eab3feba64","Type":"ContainerStarted","Data":"eb99a175ca6db3b3bbf69e3e77a59b923b3b99e2a6d831524f777aab4e65da62"}
Sep 17 18:01:56.432900 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:56.432876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" event={"ID":"a930b54c-e0da-4531-8da3-739d44c467a9","Type":"ContainerStarted","Data":"5939f1aeec2f98c0963907f7c409c921148e54379700a434a81a2bc5ba8ad5e8"}
Sep 17 18:01:56.433010 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:56.432939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" event={"ID":"a930b54c-e0da-4531-8da3-739d44c467a9","Type":"ContainerStarted","Data":"ba02fce5dc14de46010577ca3635b6c9eb59226f13deeccb2f4342e54110d68f"}
Sep 17 18:01:59.448463 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:59.448426    2566 generic.go:356] "Generic (PLEG): container finished" podID="a930b54c-e0da-4531-8da3-739d44c467a9" containerID="5939f1aeec2f98c0963907f7c409c921148e54379700a434a81a2bc5ba8ad5e8" exitCode=0
Sep 17 18:01:59.448973 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:59.448503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" event={"ID":"a930b54c-e0da-4531-8da3-739d44c467a9","Type":"ContainerDied","Data":"5939f1aeec2f98c0963907f7c409c921148e54379700a434a81a2bc5ba8ad5e8"}
Sep 17 18:01:59.450130 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:59.450109    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerID="ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a" exitCode=0
Sep 17 18:01:59.450237 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:01:59.450178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" event={"ID":"e9a7eb37-3927-420a-85ce-48eab3feba64","Type":"ContainerDied","Data":"ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a"}
Sep 17 18:02:00.457361 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:00.457321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" event={"ID":"a930b54c-e0da-4531-8da3-739d44c467a9","Type":"ContainerStarted","Data":"6d9cab93af30690b4df7609ae59dda6dd6e213719e06ccd42f0e22b0a660b2eb"}
Sep 17 18:02:00.457897 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:00.457592    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:02:00.459269 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:00.459242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" event={"ID":"e9a7eb37-3927-420a-85ce-48eab3feba64","Type":"ContainerStarted","Data":"33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43"}
Sep 17 18:02:00.476265 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:00.476214    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" podStartSLOduration=2.959711973 podStartE2EDuration="5.476199948s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.516487975s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:00.474501087 +0000 UTC m=+1680.334092468" watchObservedRunningTime="2026-09-17 18:02:00.476199948 +0000 UTC m=+1680.335791304"
Sep 17 18:02:00.494445 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:00.494394    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" podStartSLOduration=3.029922729 podStartE2EDuration="5.494378302s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.464455573s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:00.492461568 +0000 UTC m=+1680.352052936" watchObservedRunningTime="2026-09-17 18:02:00.494378302 +0000 UTC m=+1680.353969658"
Sep 17 18:02:00.739194 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:00.739082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:02:05.738754 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:05.738719    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:02:05.751619 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:05.751596    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:02:06.506889 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:06.506865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:02:31.479810 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:31.479781    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:02:32.865649 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:32.865614    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-69854679d-bqh9g_a930b54c-e0da-4531-8da3-739d44c467a9/main/0.log"
Sep 17 18:02:33.071397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.071355    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"]
Sep 17 18:02:33.075695 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.075665    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"]
Sep 17 18:02:33.075979 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.075885    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" podUID="a930b54c-e0da-4531-8da3-739d44c467a9" containerName="main" containerID="cri-o://6d9cab93af30690b4df7609ae59dda6dd6e213719e06ccd42f0e22b0a660b2eb" gracePeriod=30
Sep 17 18:02:33.081833 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:02:33.081803    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:02:33.081833 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:33.081833 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:33.081833 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43" cmd=["/bin/sleep","15"]
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.081851    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" message="" err=<
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:02:33.081863    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="main" containerID="cri-o://33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43"
Sep 17 18:02:33.082122 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.081895    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="main" containerID="cri-o://33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43" gracePeriod=60
Sep 17 18:02:33.518479 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.518206    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:02:33.602866 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.602836    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerID="33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43" exitCode=0
Sep 17 18:02:33.603074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.602925    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"
Sep 17 18:02:33.603074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.602930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" event={"ID":"e9a7eb37-3927-420a-85ce-48eab3feba64","Type":"ContainerDied","Data":"33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43"}
Sep 17 18:02:33.603074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.602989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" event={"ID":"e9a7eb37-3927-420a-85ce-48eab3feba64","Type":"ContainerDied","Data":"eb99a175ca6db3b3bbf69e3e77a59b923b3b99e2a6d831524f777aab4e65da62"}
Sep 17 18:02:33.603074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.603017    2566 scope.go:117] "RemoveContainer" containerID="33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43"
Sep 17 18:02:33.604906 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.604882    2566 generic.go:356] "Generic (PLEG): container finished" podID="a930b54c-e0da-4531-8da3-739d44c467a9" containerID="6d9cab93af30690b4df7609ae59dda6dd6e213719e06ccd42f0e22b0a660b2eb" exitCode=0
Sep 17 18:02:33.605054 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.604945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" event={"ID":"a930b54c-e0da-4531-8da3-739d44c467a9","Type":"ContainerDied","Data":"6d9cab93af30690b4df7609ae59dda6dd6e213719e06ccd42f0e22b0a660b2eb"}
Sep 17 18:02:33.612039 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612017    2566 scope.go:117] "RemoveContainer" containerID="ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a"
Sep 17 18:02:33.612209 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-home\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612294 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7eb37-3927-420a-85ce-48eab3feba64-tls-certs\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612452 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612320    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9a7eb37-3927-420a-85ce-48eab3feba64-cabundle-cert\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612452 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-dshm\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612641 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612582    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-kserve-provision-location\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612711 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-model-cache\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612711 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612646    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-home" (OuterVolumeSpecName: "home") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.612711 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-tmp-dir\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612722    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xct72\" (UniqueName: \"kubernetes.io/projected/e9a7eb37-3927-420a-85ce-48eab3feba64-kube-api-access-xct72\") pod \"e9a7eb37-3927-420a-85ce-48eab3feba64\" (UID: \"e9a7eb37-3927-420a-85ce-48eab3feba64\") "
Sep 17 18:02:33.612860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.612738    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a7eb37-3927-420a-85ce-48eab3feba64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:33.613045 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.613031    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.613099 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.613053    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9a7eb37-3927-420a-85ce-48eab3feba64-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.613386 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.613312    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.613386 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.613331    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.615069 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.615044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a7eb37-3927-420a-85ce-48eab3feba64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:33.615069 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.615059    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-dshm" (OuterVolumeSpecName: "dshm") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.615192 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.615090    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a7eb37-3927-420a-85ce-48eab3feba64-kube-api-access-xct72" (OuterVolumeSpecName: "kube-api-access-xct72") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "kube-api-access-xct72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:33.632352 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.632334    2566 scope.go:117] "RemoveContainer" containerID="33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43"
Sep 17 18:02:33.632619 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:02:33.632594    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43\": container with ID starting with 33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43 not found: ID does not exist" containerID="33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43"
Sep 17 18:02:33.632712 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.632628    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43"} err="failed to get container status \"33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43\": rpc error: code = NotFound desc = could not find container \"33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43\": container with ID starting with 33420fb9d7bc4687c3eebf6fa43ab338cc76a745fddcc25b337367cd8b181c43 not found: ID does not exist"
Sep 17 18:02:33.632712 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.632645    2566 scope.go:117] "RemoveContainer" containerID="ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a"
Sep 17 18:02:33.632942 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:02:33.632895    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a\": container with ID starting with ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a not found: ID does not exist" containerID="ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a"
Sep 17 18:02:33.633024 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.632950    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a"} err="failed to get container status \"ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a\": rpc error: code = NotFound desc = could not find container \"ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a\": container with ID starting with ef2d60d868cf69bf6b1aa7d17239a34a83dd45aefe9996cfe7f34d9cc63ba08a not found: ID does not exist"
Sep 17 18:02:33.634859 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.634837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9a7eb37-3927-420a-85ce-48eab3feba64" (UID: "e9a7eb37-3927-420a-85ce-48eab3feba64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.635395 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.635377    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:02:33.714074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.713990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-tmp-dir\") pod \"a930b54c-e0da-4531-8da3-739d44c467a9\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") "
Sep 17 18:02:33.714074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a930b54c-e0da-4531-8da3-739d44c467a9-cabundle-cert\") pod \"a930b54c-e0da-4531-8da3-739d44c467a9\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") "
Sep 17 18:02:33.714074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714052    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-kserve-provision-location\") pod \"a930b54c-e0da-4531-8da3-739d44c467a9\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") "
Sep 17 18:02:33.714362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a930b54c-e0da-4531-8da3-739d44c467a9-tls-certs\") pod \"a930b54c-e0da-4531-8da3-739d44c467a9\" (UID: \"a930b54c-e0da-4531-8da3-739d44c467a9\") "
Sep 17 18:02:33.714472 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714413    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.714472 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714433    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.714472 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714430    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a930b54c-e0da-4531-8da3-739d44c467a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a930b54c-e0da-4531-8da3-739d44c467a9" (UID: "a930b54c-e0da-4531-8da3-739d44c467a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:33.714472 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714447    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.714472 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714463    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7eb37-3927-420a-85ce-48eab3feba64-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.714472 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714475    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xct72\" (UniqueName: \"kubernetes.io/projected/e9a7eb37-3927-420a-85ce-48eab3feba64-kube-api-access-xct72\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.714691 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714485    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7eb37-3927-420a-85ce-48eab3feba64-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.714691 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.714489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a930b54c-e0da-4531-8da3-739d44c467a9" (UID: "a930b54c-e0da-4531-8da3-739d44c467a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.716246 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.716227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a930b54c-e0da-4531-8da3-739d44c467a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a930b54c-e0da-4531-8da3-739d44c467a9" (UID: "a930b54c-e0da-4531-8da3-739d44c467a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:33.737078 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.737037    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a930b54c-e0da-4531-8da3-739d44c467a9" (UID: "a930b54c-e0da-4531-8da3-739d44c467a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.815491 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.815452    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.815491 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.815484    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a930b54c-e0da-4531-8da3-739d44c467a9-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.815491 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.815494    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a930b54c-e0da-4531-8da3-739d44c467a9-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.815713 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.815503    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a930b54c-e0da-4531-8da3-739d44c467a9-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.927389 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.927360    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"]
Sep 17 18:02:33.934325 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:33.934303    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6"]
Sep 17 18:02:34.491622 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.491530    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-mx6v6" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:02:34.506674 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.506649    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" path="/var/lib/kubelet/pods/e9a7eb37-3927-420a-85ce-48eab3feba64/volumes"
Sep 17 18:02:34.610202 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.610174    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"
Sep 17 18:02:34.610202 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.610190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g" event={"ID":"a930b54c-e0da-4531-8da3-739d44c467a9","Type":"ContainerDied","Data":"ba02fce5dc14de46010577ca3635b6c9eb59226f13deeccb2f4342e54110d68f"}
Sep 17 18:02:34.610448 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.610286    2566 scope.go:117] "RemoveContainer" containerID="6d9cab93af30690b4df7609ae59dda6dd6e213719e06ccd42f0e22b0a660b2eb"
Sep 17 18:02:34.618210 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.618192    2566 scope.go:117] "RemoveContainer" containerID="5939f1aeec2f98c0963907f7c409c921148e54379700a434a81a2bc5ba8ad5e8"
Sep 17 18:02:34.647339 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.647315    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"]
Sep 17 18:02:34.648656 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:34.648637    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69854679d-bqh9g"]
Sep 17 18:02:36.506264 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:02:36.506233    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a930b54c-e0da-4531-8da3-739d44c467a9" path="/var/lib/kubelet/pods/a930b54c-e0da-4531-8da3-739d44c467a9/volumes"
Sep 17 18:03:34.398627 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.398590    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"]
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399012    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a930b54c-e0da-4531-8da3-739d44c467a9" containerName="storage-initializer"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399027    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a930b54c-e0da-4531-8da3-739d44c467a9" containerName="storage-initializer"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399038    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a930b54c-e0da-4531-8da3-739d44c467a9" containerName="main"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399046    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a930b54c-e0da-4531-8da3-739d44c467a9" containerName="main"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399065    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="main"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399071    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="main"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399083    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="storage-initializer"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399088    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="storage-initializer"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399159    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a930b54c-e0da-4531-8da3-739d44c467a9" containerName="main"
Sep 17 18:03:34.399223 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.399169    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a7eb37-3927-420a-85ce-48eab3feba64" containerName="main"
Sep 17 18:03:34.401094 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.401076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.403557 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.403527    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-bwr65\""
Sep 17 18:03:34.404702 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.404679    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 18:03:34.404832 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.404746    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:03:34.404989 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.404975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:03:34.412104 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.412080    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"]
Sep 17 18:03:34.426994 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.426895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.426994 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.426970    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tzw\" (UniqueName: \"kubernetes.io/projected/3f1d9519-cea6-4e02-9fcb-296f7cf01418-kube-api-access-b7tzw\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.427170 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.427056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.427170 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.427090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.427170 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.427151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.427267 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.427198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.527846 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.527796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.527846 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.527854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.528133 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.527892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tzw\" (UniqueName: \"kubernetes.io/projected/3f1d9519-cea6-4e02-9fcb-296f7cf01418-kube-api-access-b7tzw\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.528133 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.527969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.528133 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.528003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.528133 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.528047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.528360 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.528337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.528424 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.528397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.528531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.528422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.530347 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.530322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.530463 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.530390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.536684 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.536660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tzw\" (UniqueName: \"kubernetes.io/projected/3f1d9519-cea6-4e02-9fcb-296f7cf01418-kube-api-access-b7tzw\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.714961 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.714850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:34.987882 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:34.987848    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"]
Sep 17 18:03:34.990075 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:03:34.990043    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1d9519_cea6_4e02_9fcb_296f7cf01418.slice/crio-76ad997bd09ccea7f85024741e976b44c524e5efd9ed1074fc6a8f816a968d3a WatchSource:0}: Error finding container 76ad997bd09ccea7f85024741e976b44c524e5efd9ed1074fc6a8f816a968d3a: Status 404 returned error can't find the container with id 76ad997bd09ccea7f85024741e976b44c524e5efd9ed1074fc6a8f816a968d3a
Sep 17 18:03:35.862205 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:35.862167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" event={"ID":"3f1d9519-cea6-4e02-9fcb-296f7cf01418","Type":"ContainerStarted","Data":"3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e"}
Sep 17 18:03:35.862591 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:35.862223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" event={"ID":"3f1d9519-cea6-4e02-9fcb-296f7cf01418","Type":"ContainerStarted","Data":"76ad997bd09ccea7f85024741e976b44c524e5efd9ed1074fc6a8f816a968d3a"}
Sep 17 18:03:35.879769 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:35.879721    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" podStartSLOduration=1.8797058070000001 podStartE2EDuration="1.879705807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:35.878414317 +0000 UTC m=+1775.738005673" watchObservedRunningTime="2026-09-17 18:03:35.879705807 +0000 UTC m=+1775.739297163"
Sep 17 18:03:36.715515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:36.715468    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:44.715537 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:44.715504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:44.728403 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:44.728378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:03:44.912458 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:03:44.912420    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:04:00.181543 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.181508    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"]
Sep 17 18:04:00.184027 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.183999    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.187755 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.187719    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 18:04:00.187884 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.187723    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-9kf7l\""
Sep 17 18:04:00.187884 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.187798    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 18:04:00.187884 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.187873    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-zfdnv\""
Sep 17 18:04:00.188129 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.188113    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 18:04:00.194482 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.194462    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"]
Sep 17 18:04:00.241657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.241620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kndt\" (UniqueName: \"kubernetes.io/projected/885abeff-21e2-4be5-8906-2c296e163372-kube-api-access-9kndt\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.241869 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.241681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885abeff-21e2-4be5-8906-2c296e163372-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.343233 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.343191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kndt\" (UniqueName: \"kubernetes.io/projected/885abeff-21e2-4be5-8906-2c296e163372-kube-api-access-9kndt\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.343441 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.343359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885abeff-21e2-4be5-8906-2c296e163372-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.346227 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.346195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885abeff-21e2-4be5-8906-2c296e163372-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.353819 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.353784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kndt\" (UniqueName: \"kubernetes.io/projected/885abeff-21e2-4be5-8906-2c296e163372-kube-api-access-9kndt\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.495012 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.494893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.772156 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.772126    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"]
Sep 17 18:04:00.774314 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:04:00.774287    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885abeff_21e2_4be5_8906_2c296e163372.slice/crio-04aa9309a0ce9238042a90791b019df72a623eeaf90ccd66b5d5f4cb3cfff26f WatchSource:0}: Error finding container 04aa9309a0ce9238042a90791b019df72a623eeaf90ccd66b5d5f4cb3cfff26f: Status 404 returned error can't find the container with id 04aa9309a0ce9238042a90791b019df72a623eeaf90ccd66b5d5f4cb3cfff26f
Sep 17 18:04:00.968599 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.968561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" event={"ID":"885abeff-21e2-4be5-8906-2c296e163372","Type":"ContainerStarted","Data":"2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7"}
Sep 17 18:04:00.968810 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.968622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" event={"ID":"885abeff-21e2-4be5-8906-2c296e163372","Type":"ContainerStarted","Data":"04aa9309a0ce9238042a90791b019df72a623eeaf90ccd66b5d5f4cb3cfff26f"}
Sep 17 18:04:00.968810 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.968648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:00.993865 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:00.993812    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" podStartSLOduration=0.993797913 podStartE2EDuration="993.797913ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:00.991988634 +0000 UTC m=+1800.851579990" watchObservedRunningTime="2026-09-17 18:04:00.993797913 +0000 UTC m=+1800.853389268"
Sep 17 18:04:02.527266 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:02.527235    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:04:02.529775 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:02.529747    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:04:31.975196 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:31.975163    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:04:34.065519 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:34.065482    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"]
Sep 17 18:04:41.975017 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:41.974980    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:49.084462 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:49.084414    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main" containerID="cri-o://2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7" gracePeriod=45
Sep 17 18:04:51.557669 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:51.557638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"]
Sep 17 18:04:51.566381 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:04:51.566348    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:04:51.566381 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:51.566381 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:51.566381 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e" cmd=["/bin/sleep","15"]
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:51.566396    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" message="" err=<
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:04:51.566408    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" podUID="3f1d9519-cea6-4e02-9fcb-296f7cf01418" containerName="main" containerID="cri-o://3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e"
Sep 17 18:04:51.566634 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:51.566450    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" podUID="3f1d9519-cea6-4e02-9fcb-296f7cf01418" containerName="main" containerID="cri-o://3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e" gracePeriod=60
Sep 17 18:04:51.903560 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:51.903532    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:04:51.975275 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:51.975239    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:52.009580 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.009559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-model-cache\") pod \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") "
Sep 17 18:04:52.009702 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.009604    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tls-certs\") pod \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") "
Sep 17 18:04:52.009702 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.009635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7tzw\" (UniqueName: \"kubernetes.io/projected/3f1d9519-cea6-4e02-9fcb-296f7cf01418-kube-api-access-b7tzw\") pod \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") "
Sep 17 18:04:52.009702 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.009651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-dshm\") pod \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") "
Sep 17 18:04:52.009702 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.009674    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tmp-dir\") pod \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") "
Sep 17 18:04:52.009904 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.009723    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-home\") pod \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\" (UID: \"3f1d9519-cea6-4e02-9fcb-296f7cf01418\") "
Sep 17 18:04:52.009904 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.009817    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f1d9519-cea6-4e02-9fcb-296f7cf01418" (UID: "3f1d9519-cea6-4e02-9fcb-296f7cf01418"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:52.010055 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.010013    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.010055 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.010012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f1d9519-cea6-4e02-9fcb-296f7cf01418" (UID: "3f1d9519-cea6-4e02-9fcb-296f7cf01418"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:52.010055 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.010044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-home" (OuterVolumeSpecName: "home") pod "3f1d9519-cea6-4e02-9fcb-296f7cf01418" (UID: "3f1d9519-cea6-4e02-9fcb-296f7cf01418"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:52.011966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.011940    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-dshm" (OuterVolumeSpecName: "dshm") pod "3f1d9519-cea6-4e02-9fcb-296f7cf01418" (UID: "3f1d9519-cea6-4e02-9fcb-296f7cf01418"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:52.012075 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.012032    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1d9519-cea6-4e02-9fcb-296f7cf01418-kube-api-access-b7tzw" (OuterVolumeSpecName: "kube-api-access-b7tzw") pod "3f1d9519-cea6-4e02-9fcb-296f7cf01418" (UID: "3f1d9519-cea6-4e02-9fcb-296f7cf01418"). InnerVolumeSpecName "kube-api-access-b7tzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:52.012075 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.012032    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f1d9519-cea6-4e02-9fcb-296f7cf01418" (UID: "3f1d9519-cea6-4e02-9fcb-296f7cf01418"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:52.110640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.110563    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.110640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.110589    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.110640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.110599    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tzw\" (UniqueName: \"kubernetes.io/projected/3f1d9519-cea6-4e02-9fcb-296f7cf01418-kube-api-access-b7tzw\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.110640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.110608    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.110640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.110616    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1d9519-cea6-4e02-9fcb-296f7cf01418-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.192213 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.192179    2566 generic.go:356] "Generic (PLEG): container finished" podID="3f1d9519-cea6-4e02-9fcb-296f7cf01418" containerID="3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e" exitCode=0
Sep 17 18:04:52.192384 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.192230    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"
Sep 17 18:04:52.192384 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.192270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" event={"ID":"3f1d9519-cea6-4e02-9fcb-296f7cf01418","Type":"ContainerDied","Data":"3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e"}
Sep 17 18:04:52.192384 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.192325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n" event={"ID":"3f1d9519-cea6-4e02-9fcb-296f7cf01418","Type":"ContainerDied","Data":"76ad997bd09ccea7f85024741e976b44c524e5efd9ed1074fc6a8f816a968d3a"}
Sep 17 18:04:52.192384 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.192347    2566 scope.go:117] "RemoveContainer" containerID="3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e"
Sep 17 18:04:52.201220 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.201205    2566 scope.go:117] "RemoveContainer" containerID="3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e"
Sep 17 18:04:52.201488 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:04:52.201468    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e\": container with ID starting with 3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e not found: ID does not exist" containerID="3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e"
Sep 17 18:04:52.201561 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.201497    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e"} err="failed to get container status \"3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e\": rpc error: code = NotFound desc = could not find container \"3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e\": container with ID starting with 3e42c9b6ac9335b5adc58bc3fb106f1a81cc79061af2d7521401d3a1a692093e not found: ID does not exist"
Sep 17 18:04:52.224345 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.224317    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"]
Sep 17 18:04:52.231625 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.231602    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972x86n"]
Sep 17 18:04:52.511670 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:04:52.511592    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f1d9519-cea6-4e02-9fcb-296f7cf01418" path="/var/lib/kubelet/pods/3f1d9519-cea6-4e02-9fcb-296f7cf01418/volumes"
Sep 17 18:05:01.975514 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:01.975473    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:05:01.977069 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:01.975581    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:05:03.915745 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.915710    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"]
Sep 17 18:05:03.916268 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.916249    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f1d9519-cea6-4e02-9fcb-296f7cf01418" containerName="main"
Sep 17 18:05:03.916346 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.916271    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f1d9519-cea6-4e02-9fcb-296f7cf01418" containerName="main"
Sep 17 18:05:03.916403 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.916370    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f1d9519-cea6-4e02-9fcb-296f7cf01418" containerName="main"
Sep 17 18:05:03.919391 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.919370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:03.922081 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.922057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:05:03.922212 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.922107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-9vpd5\""
Sep 17 18:05:03.922212 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.922131    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 18:05:03.922212 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.922169    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:05:03.922368 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.922348    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:05:03.922423 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.922373    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:05:03.937476 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:03.937453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"]
Sep 17 18:05:04.018397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.018397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.018605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kswfb\" (UniqueName: \"kubernetes.io/projected/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kube-api-access-kswfb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.018605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.018605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.018605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.018740 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.018740 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.018654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119456 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswfb\" (UniqueName: \"kubernetes.io/projected/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kube-api-access-kswfb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119938 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.119938 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.119686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.120090 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.120061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.120157 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.120101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.120210 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.120155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.120210 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.120178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.120769 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.120745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.121885 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.121859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.122224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.122207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.139363 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.139334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kswfb\" (UniqueName: \"kubernetes.io/projected/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kube-api-access-kswfb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.229459 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.229377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:04.511398 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:04.511311    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"]
Sep 17 18:05:04.514586 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:05:04.514560    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f6fbb2_5d3b_4f09_af90_b6e0506b2e2c.slice/crio-6b9614da573ad99f40d7d1679f8f2f7a2d37fab6ca2de8c2ac7bcb7a7907f33a WatchSource:0}: Error finding container 6b9614da573ad99f40d7d1679f8f2f7a2d37fab6ca2de8c2ac7bcb7a7907f33a: Status 404 returned error can't find the container with id 6b9614da573ad99f40d7d1679f8f2f7a2d37fab6ca2de8c2ac7bcb7a7907f33a
Sep 17 18:05:05.249146 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:05.249111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" event={"ID":"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c","Type":"ContainerStarted","Data":"ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943"}
Sep 17 18:05:05.249530 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:05.249174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" event={"ID":"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c","Type":"ContainerStarted","Data":"6b9614da573ad99f40d7d1679f8f2f7a2d37fab6ca2de8c2ac7bcb7a7907f33a"}
Sep 17 18:05:07.259392 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:07.259357    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerID="ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943" exitCode=0
Sep 17 18:05:07.259775 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:07.259431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" event={"ID":"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c","Type":"ContainerDied","Data":"ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943"}
Sep 17 18:05:07.260830 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:07.260815    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:05:11.975170 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:11.975126    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:05:20.216126 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.216096    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:05:20.281144 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.281107    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885abeff-21e2-4be5-8906-2c296e163372-tls-certs\") pod \"885abeff-21e2-4be5-8906-2c296e163372\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") "
Sep 17 18:05:20.281335 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.281227    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kndt\" (UniqueName: \"kubernetes.io/projected/885abeff-21e2-4be5-8906-2c296e163372-kube-api-access-9kndt\") pod \"885abeff-21e2-4be5-8906-2c296e163372\" (UID: \"885abeff-21e2-4be5-8906-2c296e163372\") "
Sep 17 18:05:20.284170 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.284126    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/885abeff-21e2-4be5-8906-2c296e163372-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "885abeff-21e2-4be5-8906-2c296e163372" (UID: "885abeff-21e2-4be5-8906-2c296e163372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:20.284320 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.284203    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885abeff-21e2-4be5-8906-2c296e163372-kube-api-access-9kndt" (OuterVolumeSpecName: "kube-api-access-9kndt") pod "885abeff-21e2-4be5-8906-2c296e163372" (UID: "885abeff-21e2-4be5-8906-2c296e163372"). InnerVolumeSpecName "kube-api-access-9kndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:20.349133 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.349096    2566 generic.go:356] "Generic (PLEG): container finished" podID="885abeff-21e2-4be5-8906-2c296e163372" containerID="2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7" exitCode=0
Sep 17 18:05:20.349314 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.349158    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"
Sep 17 18:05:20.349314 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.349192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" event={"ID":"885abeff-21e2-4be5-8906-2c296e163372","Type":"ContainerDied","Data":"2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7"}
Sep 17 18:05:20.349314 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.349258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6" event={"ID":"885abeff-21e2-4be5-8906-2c296e163372","Type":"ContainerDied","Data":"04aa9309a0ce9238042a90791b019df72a623eeaf90ccd66b5d5f4cb3cfff26f"}
Sep 17 18:05:20.349442 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.349354    2566 scope.go:117] "RemoveContainer" containerID="2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7"
Sep 17 18:05:20.360039 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.360016    2566 scope.go:117] "RemoveContainer" containerID="2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7"
Sep 17 18:05:20.360399 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:05:20.360350    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7\": container with ID starting with 2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7 not found: ID does not exist" containerID="2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7"
Sep 17 18:05:20.360480 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.360422    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7"} err="failed to get container status \"2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7\": rpc error: code = NotFound desc = could not find container \"2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7\": container with ID starting with 2dea7405de36326d2eec4c8812ff013554183e04311a66bed1538f67b5d998d7 not found: ID does not exist"
Sep 17 18:05:20.376878 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.376812    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"]
Sep 17 18:05:20.382777 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.382747    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885abeff-21e2-4be5-8906-2c296e163372-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:05:20.382777 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.382780    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kndt\" (UniqueName: \"kubernetes.io/projected/885abeff-21e2-4be5-8906-2c296e163372-kube-api-access-9kndt\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:05:20.383438 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.383416    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen8zf6"]
Sep 17 18:05:20.508044 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:20.507961    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="885abeff-21e2-4be5-8906-2c296e163372" path="/var/lib/kubelet/pods/885abeff-21e2-4be5-8906-2c296e163372/volumes"
Sep 17 18:05:35.424295 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:35.424256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" event={"ID":"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c","Type":"ContainerStarted","Data":"d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6"}
Sep 17 18:05:35.448809 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:35.448587    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podStartSLOduration=2.25851439 podStartE2EDuration="32.448565751s" totalImagesPullingTime="28.011214696s" totalInitContainerRuntime="2.178836665s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:35.447229743 +0000 UTC m=+1895.306821124" watchObservedRunningTime="2026-09-17 18:05:35.448565751 +0000 UTC m=+1895.308157107"
Sep 17 18:05:36.229992 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:36.229950    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:44.230098 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:44.230054    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:05:44.231743 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:44.231711    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:05:54.230291 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:05:54.230251    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:06:04.229973 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:04.229899    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:06:11.549283 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.549247    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"]
Sep 17 18:06:11.549758 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.549572    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main"
Sep 17 18:06:11.549758 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.549583    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main"
Sep 17 18:06:11.549758 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.549647    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="885abeff-21e2-4be5-8906-2c296e163372" containerName="main"
Sep 17 18:06:11.556397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.556372    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:11.560209 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.560171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-ccf99\""
Sep 17 18:06:11.560209 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.560198    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:06:11.560432 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.560331    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-cj5jk\""
Sep 17 18:06:11.560432 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.560399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:06:11.560548 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.560442    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:06:11.565795 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.565773    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"]
Sep 17 18:06:11.663142 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.663102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24e8f62-53b7-4dd6-b248-42287ca31257-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:11.663319 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.663240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blsmf\" (UniqueName: \"kubernetes.io/projected/e24e8f62-53b7-4dd6-b248-42287ca31257-kube-api-access-blsmf\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:11.764149 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.764107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blsmf\" (UniqueName: \"kubernetes.io/projected/e24e8f62-53b7-4dd6-b248-42287ca31257-kube-api-access-blsmf\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:11.764302 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.764174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24e8f62-53b7-4dd6-b248-42287ca31257-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:11.766766 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.766732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24e8f62-53b7-4dd6-b248-42287ca31257-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:11.773598 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.773574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blsmf\" (UniqueName: \"kubernetes.io/projected/e24e8f62-53b7-4dd6-b248-42287ca31257-kube-api-access-blsmf\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:11.868401 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:11.868292    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:12.152837 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:12.152811    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"]
Sep 17 18:06:12.156144 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:06:12.156110    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24e8f62_53b7_4dd6_b248_42287ca31257.slice/crio-2bb2e2b7704a2a79fc8eb28cb8b7b8b42a338cdcc9601f1e316ca5309f39e5ca WatchSource:0}: Error finding container 2bb2e2b7704a2a79fc8eb28cb8b7b8b42a338cdcc9601f1e316ca5309f39e5ca: Status 404 returned error can't find the container with id 2bb2e2b7704a2a79fc8eb28cb8b7b8b42a338cdcc9601f1e316ca5309f39e5ca
Sep 17 18:06:12.607219 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:12.607179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" event={"ID":"e24e8f62-53b7-4dd6-b248-42287ca31257","Type":"ContainerStarted","Data":"2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6"}
Sep 17 18:06:12.607674 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:12.607234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" event={"ID":"e24e8f62-53b7-4dd6-b248-42287ca31257","Type":"ContainerStarted","Data":"2bb2e2b7704a2a79fc8eb28cb8b7b8b42a338cdcc9601f1e316ca5309f39e5ca"}
Sep 17 18:06:12.607674 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:12.607290    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:12.627723 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:12.627664    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" podStartSLOduration=1.6276448989999999 podStartE2EDuration="1.627644899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:12.626671575 +0000 UTC m=+1932.486262925" watchObservedRunningTime="2026-09-17 18:06:12.627644899 +0000 UTC m=+1932.487236254"
Sep 17 18:06:14.230357 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:14.230313    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:06:24.230313 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:24.230265    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:06:34.230280 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:34.230220    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:06:43.613450 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:43.613360    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:06:44.230395 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:44.230346    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:06:54.230864 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:06:54.230809    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:07:04.230471 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:04.230424    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:07:14.230477 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:14.230421    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:07:24.230129 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:24.230081    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:07:34.239933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:34.239881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:07:34.247632 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:34.247613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:07:39.013646 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:39.013613    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"]
Sep 17 18:07:54.035410 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.035346    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" containerID="cri-o://d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6" gracePeriod=45
Sep 17 18:07:54.241363 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.241316    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:07:54.687931 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.687894    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:07:54.693496 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-home\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.693595 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tls-certs\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.693595 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-dshm\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.693595 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693590    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kserve-provision-location\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.693754 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693617    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-cabundle-cert\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.693754 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693634    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tmp-dir\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.693754 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693648    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-model-cache\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.693754 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.693680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kswfb\" (UniqueName: \"kubernetes.io/projected/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kube-api-access-kswfb\") pod \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\" (UID: \"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c\") "
Sep 17 18:07:54.694048 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.694022    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:54.694409 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.694258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-home" (OuterVolumeSpecName: "home") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:54.694522 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.694424    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:54.696326 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.696295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:54.696326 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.696302    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-dshm" (OuterVolumeSpecName: "dshm") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:54.696484 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.696408    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kube-api-access-kswfb" (OuterVolumeSpecName: "kube-api-access-kswfb") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "kube-api-access-kswfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:54.706289 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.706265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:54.718763 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.718733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" (UID: "e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:54.794317 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794254    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:54.794317 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794299    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:54.794317 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794308    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:54.794317 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794316    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kswfb\" (UniqueName: \"kubernetes.io/projected/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kube-api-access-kswfb\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:54.794515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794326    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:54.794515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794333    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:54.794515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794342    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:54.794515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:54.794349    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:07:55.063527 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.063490    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerID="d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6" exitCode=0
Sep 17 18:07:55.063957 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.063537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" event={"ID":"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c","Type":"ContainerDied","Data":"d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6"}
Sep 17 18:07:55.063957 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.063544    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"
Sep 17 18:07:55.063957 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.063579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m" event={"ID":"e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c","Type":"ContainerDied","Data":"6b9614da573ad99f40d7d1679f8f2f7a2d37fab6ca2de8c2ac7bcb7a7907f33a"}
Sep 17 18:07:55.063957 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.063600    2566 scope.go:117] "RemoveContainer" containerID="d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6"
Sep 17 18:07:55.072571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.072556    2566 scope.go:117] "RemoveContainer" containerID="ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943"
Sep 17 18:07:55.079846 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.079826    2566 scope.go:117] "RemoveContainer" containerID="d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6"
Sep 17 18:07:55.080131 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:07:55.080112    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6\": container with ID starting with d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6 not found: ID does not exist" containerID="d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6"
Sep 17 18:07:55.080189 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.080139    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6"} err="failed to get container status \"d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6\": rpc error: code = NotFound desc = could not find container \"d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6\": container with ID starting with d19d093ad0fd520a387f5a8ea1c61c1b938ccff609757ddeca5456c90c06dfa6 not found: ID does not exist"
Sep 17 18:07:55.080189 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.080155    2566 scope.go:117] "RemoveContainer" containerID="ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943"
Sep 17 18:07:55.080377 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:07:55.080357    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943\": container with ID starting with ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943 not found: ID does not exist" containerID="ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943"
Sep 17 18:07:55.080436 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.080387    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943"} err="failed to get container status \"ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943\": rpc error: code = NotFound desc = could not find container \"ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943\": container with ID starting with ddfc34590b6c80cccae7e07b266b6a1fa550f018a02b770ef17f18f594265943 not found: ID does not exist"
Sep 17 18:07:55.088220 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.088198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"]
Sep 17 18:07:55.095281 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:55.095260    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6ff46898c5nqv4m"]
Sep 17 18:07:56.506402 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:07:56.506368    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" path="/var/lib/kubelet/pods/e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c/volumes"
Sep 17 18:08:19.034778 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:19.034702    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"]
Sep 17 18:08:23.614113 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:23.614071    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:32.875294 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.875255    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"]
Sep 17 18:08:32.875839 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.875607    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main"
Sep 17 18:08:32.875839 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.875619    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main"
Sep 17 18:08:32.875839 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.875630    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="storage-initializer"
Sep 17 18:08:32.875839 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.875636    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="storage-initializer"
Sep 17 18:08:32.875839 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.875696    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f6fbb2-5d3b-4f09-af90-b6e0506b2e2c" containerName="main"
Sep 17 18:08:32.878639 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.878619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:32.881217 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.881199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 18:08:32.882477 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.882455    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-k5jlq\""
Sep 17 18:08:32.882581 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.882486    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 18:08:32.882581 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.882486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-klzqs\""
Sep 17 18:08:32.882673 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.882460    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 18:08:32.888166 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.888147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"]
Sep 17 18:08:32.996535 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.996508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkm2c\" (UniqueName: \"kubernetes.io/projected/278dcd2e-6f25-42e4-b7ab-73d481eb728f-kube-api-access-tkm2c\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:32.996717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:32.996556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278dcd2e-6f25-42e4-b7ab-73d481eb728f-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:33.097570 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:33.097529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkm2c\" (UniqueName: \"kubernetes.io/projected/278dcd2e-6f25-42e4-b7ab-73d481eb728f-kube-api-access-tkm2c\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:33.097762 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:33.097589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278dcd2e-6f25-42e4-b7ab-73d481eb728f-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:33.100355 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:33.100330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278dcd2e-6f25-42e4-b7ab-73d481eb728f-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:33.106700 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:33.106675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkm2c\" (UniqueName: \"kubernetes.io/projected/278dcd2e-6f25-42e4-b7ab-73d481eb728f-kube-api-access-tkm2c\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:33.190677 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:33.190591    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:33.461242 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:33.461215    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"]
Sep 17 18:08:33.464728 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:08:33.464701    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278dcd2e_6f25_42e4_b7ab_73d481eb728f.slice/crio-bbda26bd635b023d588cfdda51302d8a5641f4496a1e2095ac2744158b73c527 WatchSource:0}: Error finding container bbda26bd635b023d588cfdda51302d8a5641f4496a1e2095ac2744158b73c527: Status 404 returned error can't find the container with id bbda26bd635b023d588cfdda51302d8a5641f4496a1e2095ac2744158b73c527
Sep 17 18:08:33.613788 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:33.613751    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:34.053762 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:34.053719    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main" containerID="cri-o://2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6" gracePeriod=45
Sep 17 18:08:34.237616 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:34.237575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" event={"ID":"278dcd2e-6f25-42e4-b7ab-73d481eb728f","Type":"ContainerStarted","Data":"a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab"}
Sep 17 18:08:34.237825 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:34.237656    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" event={"ID":"278dcd2e-6f25-42e4-b7ab-73d481eb728f","Type":"ContainerStarted","Data":"bbda26bd635b023d588cfdda51302d8a5641f4496a1e2095ac2744158b73c527"}
Sep 17 18:08:34.237825 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:34.237676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:08:34.260247 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:34.260187    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" podStartSLOduration=2.260169127 podStartE2EDuration="2.260169127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:34.259954932 +0000 UTC m=+2074.119546289" watchObservedRunningTime="2026-09-17 18:08:34.260169127 +0000 UTC m=+2074.119760484"
Sep 17 18:08:43.613148 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:43.613111    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:43.613537 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:43.613226    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:08:53.613564 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:08:53.613525    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:09:02.556717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:02.556680    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:09:02.561843 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:02.561808    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:09:03.614186 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:03.614151    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:09:04.302027 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.301995    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:09:04.367357 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.367260    2566 generic.go:356] "Generic (PLEG): container finished" podID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerID="2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6" exitCode=0
Sep 17 18:09:04.367357 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.367309    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"
Sep 17 18:09:04.367558 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.367352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" event={"ID":"e24e8f62-53b7-4dd6-b248-42287ca31257","Type":"ContainerDied","Data":"2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6"}
Sep 17 18:09:04.367558 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.367431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d" event={"ID":"e24e8f62-53b7-4dd6-b248-42287ca31257","Type":"ContainerDied","Data":"2bb2e2b7704a2a79fc8eb28cb8b7b8b42a338cdcc9601f1e316ca5309f39e5ca"}
Sep 17 18:09:04.367558 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.367459    2566 scope.go:117] "RemoveContainer" containerID="2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6"
Sep 17 18:09:04.371763 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.371738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24e8f62-53b7-4dd6-b248-42287ca31257-tls-certs\") pod \"e24e8f62-53b7-4dd6-b248-42287ca31257\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") "
Sep 17 18:09:04.371871 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.371800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blsmf\" (UniqueName: \"kubernetes.io/projected/e24e8f62-53b7-4dd6-b248-42287ca31257-kube-api-access-blsmf\") pod \"e24e8f62-53b7-4dd6-b248-42287ca31257\" (UID: \"e24e8f62-53b7-4dd6-b248-42287ca31257\") "
Sep 17 18:09:04.373976 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.373950    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24e8f62-53b7-4dd6-b248-42287ca31257-kube-api-access-blsmf" (OuterVolumeSpecName: "kube-api-access-blsmf") pod "e24e8f62-53b7-4dd6-b248-42287ca31257" (UID: "e24e8f62-53b7-4dd6-b248-42287ca31257"). InnerVolumeSpecName "kube-api-access-blsmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:04.374230 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.374206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24e8f62-53b7-4dd6-b248-42287ca31257-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e24e8f62-53b7-4dd6-b248-42287ca31257" (UID: "e24e8f62-53b7-4dd6-b248-42287ca31257"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:04.377024 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.377006    2566 scope.go:117] "RemoveContainer" containerID="2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6"
Sep 17 18:09:04.377335 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:09:04.377313    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6\": container with ID starting with 2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6 not found: ID does not exist" containerID="2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6"
Sep 17 18:09:04.377389 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.377342    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6"} err="failed to get container status \"2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6\": rpc error: code = NotFound desc = could not find container \"2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6\": container with ID starting with 2a604864d5c62061f616958ada80afbe36f00baf4953e0ad743574749e3caca6 not found: ID does not exist"
Sep 17 18:09:04.472984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.472947    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24e8f62-53b7-4dd6-b248-42287ca31257-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:09:04.472984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.472977    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blsmf\" (UniqueName: \"kubernetes.io/projected/e24e8f62-53b7-4dd6-b248-42287ca31257-kube-api-access-blsmf\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:09:04.685970 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.685868    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"]
Sep 17 18:09:04.691274 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:04.691247    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheqrq8d"]
Sep 17 18:09:05.243212 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:05.243182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:09:06.505996 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:06.505961    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" path="/var/lib/kubelet/pods/e24e8f62-53b7-4dd6-b248-42287ca31257/volumes"
Sep 17 18:09:22.718445 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.718411    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"]
Sep 17 18:09:22.719030 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.718952    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main"
Sep 17 18:09:22.719030 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.718973    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main"
Sep 17 18:09:22.719164 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.719065    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24e8f62-53b7-4dd6-b248-42287ca31257" containerName="main"
Sep 17 18:09:22.723272 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.723249    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.726210 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.726177    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:09:22.727369 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.727347    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:09:22.727496 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.727382    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-qwq9g\""
Sep 17 18:09:22.727496 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.727384    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:09:22.736286 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.736256    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"]
Sep 17 18:09:22.836483 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.836441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.836669 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.836523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.836669 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.836551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqchj\" (UniqueName: \"kubernetes.io/projected/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-kube-api-access-mqchj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.836669 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.836568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.836669 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.836589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.836812 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.836673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.937726 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.937693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.937726 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.937730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqchj\" (UniqueName: \"kubernetes.io/projected/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-kube-api-access-mqchj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.937968 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.937749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.937968 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.937865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.937968 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.937936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.938092 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.938026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.938312 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.938285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.938430 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.938292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.938430 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.938345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.940054 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.940035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.940355 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.940337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:22.952430 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:22.952397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqchj\" (UniqueName: \"kubernetes.io/projected/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-kube-api-access-mqchj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:23.037479 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:23.037389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:23.309984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:23.309960    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"]
Sep 17 18:09:23.312092 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:09:23.312067    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6c856f_b9f9_4eb8_86f2_29af72f1bb8c.slice/crio-791aee29e081ecff50fe1647c4f436c8608e57eb597b10ac9a2ec10f922bc32f WatchSource:0}: Error finding container 791aee29e081ecff50fe1647c4f436c8608e57eb597b10ac9a2ec10f922bc32f: Status 404 returned error can't find the container with id 791aee29e081ecff50fe1647c4f436c8608e57eb597b10ac9a2ec10f922bc32f
Sep 17 18:09:23.450040 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:23.449999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" event={"ID":"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c","Type":"ContainerStarted","Data":"78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd"}
Sep 17 18:09:23.450222 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:23.450057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" event={"ID":"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c","Type":"ContainerStarted","Data":"791aee29e081ecff50fe1647c4f436c8608e57eb597b10ac9a2ec10f922bc32f"}
Sep 17 18:09:23.469425 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:23.469369    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" podStartSLOduration=1.469353254 podStartE2EDuration="1.469353254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:23.467895326 +0000 UTC m=+2123.327486676" watchObservedRunningTime="2026-09-17 18:09:23.469353254 +0000 UTC m=+2123.328944609"
Sep 17 18:09:24.037803 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:24.037756    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:33.038349 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:33.038311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:33.052039 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:33.052006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:09:33.502748 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:09:33.502716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:11:18.091602 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:18.091525    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"]
Sep 17 18:11:25.243525 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:25.243481    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:33.110743 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:33.110695    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main" containerID="cri-o://a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab" gracePeriod=45
Sep 17 18:11:35.242923 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:35.242872    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:39.840394 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:39.840360    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"]
Sep 17 18:11:39.850502 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:11:39.850461    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:11:39.850502 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:39.850502 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:39.850502 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd" cmd=["/bin/sleep","15"]
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:39.850524    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" message="" err=<
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:11:39.850537    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" podUID="3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" containerName="main" containerID="cri-o://78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd"
Sep 17 18:11:39.850778 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:39.850590    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" podUID="3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" containerName="main" containerID="cri-o://78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd" gracePeriod=60
Sep 17 18:11:40.199468 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.199439    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:11:40.327261 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-home\") pod \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") "
Sep 17 18:11:40.327461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-model-cache\") pod \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") "
Sep 17 18:11:40.327461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327304    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tmp-dir\") pod \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") "
Sep 17 18:11:40.327461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqchj\" (UniqueName: \"kubernetes.io/projected/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-kube-api-access-mqchj\") pod \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") "
Sep 17 18:11:40.327461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327393    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tls-certs\") pod \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") "
Sep 17 18:11:40.327461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-dshm\") pod \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\" (UID: \"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c\") "
Sep 17 18:11:40.327729 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327590    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-home" (OuterVolumeSpecName: "home") pod "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" (UID: "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:40.327729 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327601    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" (UID: "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:40.327729 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327612    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" (UID: "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:40.327729 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.327689    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:11:40.329791 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.329756    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" (UID: "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:40.329944 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.329815    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-kube-api-access-mqchj" (OuterVolumeSpecName: "kube-api-access-mqchj") pod "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" (UID: "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c"). InnerVolumeSpecName "kube-api-access-mqchj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:40.329944 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.329845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-dshm" (OuterVolumeSpecName: "dshm") pod "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" (UID: "3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:40.429136 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.429026    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:11:40.429136 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.429073    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqchj\" (UniqueName: \"kubernetes.io/projected/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-kube-api-access-mqchj\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:11:40.429136 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.429087    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:11:40.429136 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.429098    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:11:40.429136 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:40.429109    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:11:41.017032 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.017000    2566 generic.go:356] "Generic (PLEG): container finished" podID="3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" containerID="78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd" exitCode=0
Sep 17 18:11:41.017481 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.017048    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"
Sep 17 18:11:41.017481 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.017078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" event={"ID":"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c","Type":"ContainerDied","Data":"78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd"}
Sep 17 18:11:41.017481 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.017131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f" event={"ID":"3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c","Type":"ContainerDied","Data":"791aee29e081ecff50fe1647c4f436c8608e57eb597b10ac9a2ec10f922bc32f"}
Sep 17 18:11:41.017481 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.017156    2566 scope.go:117] "RemoveContainer" containerID="78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd"
Sep 17 18:11:41.026015 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.025994    2566 scope.go:117] "RemoveContainer" containerID="78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd"
Sep 17 18:11:41.026308 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:11:41.026286    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd\": container with ID starting with 78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd not found: ID does not exist" containerID="78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd"
Sep 17 18:11:41.026355 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.026316    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd"} err="failed to get container status \"78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd\": rpc error: code = NotFound desc = could not find container \"78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd\": container with ID starting with 78f86086efcccd0fd9643084fa410b23317937ea65f505e27d185205e4a5c7bd not found: ID does not exist"
Sep 17 18:11:41.035950 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.035926    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"]
Sep 17 18:11:41.040682 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:41.040660    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969srg4f"]
Sep 17 18:11:42.507171 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:42.507139    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" path="/var/lib/kubelet/pods/3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c/volumes"
Sep 17 18:11:45.243835 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:45.243799    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:45.244264 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:45.243898    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:11:55.242965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:11:55.242897    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:03.351314 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:03.351288    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:12:03.417693 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:03.417595    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278dcd2e-6f25-42e4-b7ab-73d481eb728f-tls-certs\") pod \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") "
Sep 17 18:12:03.417693 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:03.417693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkm2c\" (UniqueName: \"kubernetes.io/projected/278dcd2e-6f25-42e4-b7ab-73d481eb728f-kube-api-access-tkm2c\") pod \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\" (UID: \"278dcd2e-6f25-42e4-b7ab-73d481eb728f\") "
Sep 17 18:12:03.424497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:03.420888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278dcd2e-6f25-42e4-b7ab-73d481eb728f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "278dcd2e-6f25-42e4-b7ab-73d481eb728f" (UID: "278dcd2e-6f25-42e4-b7ab-73d481eb728f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:03.424497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:03.420899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278dcd2e-6f25-42e4-b7ab-73d481eb728f-kube-api-access-tkm2c" (OuterVolumeSpecName: "kube-api-access-tkm2c") pod "278dcd2e-6f25-42e4-b7ab-73d481eb728f" (UID: "278dcd2e-6f25-42e4-b7ab-73d481eb728f"). InnerVolumeSpecName "kube-api-access-tkm2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:03.522362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:03.522315    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkm2c\" (UniqueName: \"kubernetes.io/projected/278dcd2e-6f25-42e4-b7ab-73d481eb728f-kube-api-access-tkm2c\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:12:03.522362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:03.522358    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278dcd2e-6f25-42e4-b7ab-73d481eb728f-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:12:04.116006 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.115975    2566 generic.go:356] "Generic (PLEG): container finished" podID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerID="a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab" exitCode=0
Sep 17 18:12:04.116187 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.116033    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"
Sep 17 18:12:04.116187 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.116060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" event={"ID":"278dcd2e-6f25-42e4-b7ab-73d481eb728f","Type":"ContainerDied","Data":"a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab"}
Sep 17 18:12:04.116187 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.116111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb" event={"ID":"278dcd2e-6f25-42e4-b7ab-73d481eb728f","Type":"ContainerDied","Data":"bbda26bd635b023d588cfdda51302d8a5641f4496a1e2095ac2744158b73c527"}
Sep 17 18:12:04.116187 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.116137    2566 scope.go:117] "RemoveContainer" containerID="a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab"
Sep 17 18:12:04.125184 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.125166    2566 scope.go:117] "RemoveContainer" containerID="a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab"
Sep 17 18:12:04.125441 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:12:04.125420    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab\": container with ID starting with a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab not found: ID does not exist" containerID="a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab"
Sep 17 18:12:04.125521 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.125454    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab"} err="failed to get container status \"a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab\": rpc error: code = NotFound desc = could not find container \"a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab\": container with ID starting with a083ea6167bc3082324d5fd0240057c40ea964b7621e28777f7e655baa1333ab not found: ID does not exist"
Sep 17 18:12:04.140007 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.139983    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"]
Sep 17 18:12:04.150670 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.150649    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheksmnb"]
Sep 17 18:12:04.506431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:04.506353    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" path="/var/lib/kubelet/pods/278dcd2e-6f25-42e4-b7ab-73d481eb728f/volumes"
Sep 17 18:12:14.823984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.823948    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"]
Sep 17 18:12:14.824351 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.824284    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main"
Sep 17 18:12:14.824351 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.824297    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main"
Sep 17 18:12:14.824351 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.824310    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" containerName="main"
Sep 17 18:12:14.824351 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.824316    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" containerName="main"
Sep 17 18:12:14.824486 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.824379    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6c856f-b9f9-4eb8-86f2-29af72f1bb8c" containerName="main"
Sep 17 18:12:14.824486 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.824389    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="278dcd2e-6f25-42e4-b7ab-73d481eb728f" containerName="main"
Sep 17 18:12:14.829000 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.828982    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.831797 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.831633    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 18:12:14.831797 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.831667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 18:12:14.831797 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.831763    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-x82bg\""
Sep 17 18:12:14.831797 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.831787    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 18:12:14.832200 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.831975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:12:14.832391 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.832379    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 18:12:14.842113 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.842090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"]
Sep 17 18:12:14.920610 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtb7\" (UniqueName: \"kubernetes.io/projected/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kube-api-access-jqtb7\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.920793 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-dshm\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.920793 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tmp-dir\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.920793 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-model-cache\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.920793 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.920986 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-home\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.920986 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-cabundle-cert\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:14.920986 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:14.920867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tls-certs\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021516 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-model-cache\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021516 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021771 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-home\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021771 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-cabundle-cert\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021771 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tls-certs\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtb7\" (UniqueName: \"kubernetes.io/projected/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kube-api-access-jqtb7\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-dshm\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tmp-dir\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.021966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.021935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-model-cache\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.022139 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.022005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.022139 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.022058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-home\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.022220 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.022134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tmp-dir\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.022334 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.022315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-cabundle-cert\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.024203 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.024180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-dshm\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.024385 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.024367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tls-certs\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.032397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.032374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtb7\" (UniqueName: \"kubernetes.io/projected/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kube-api-access-jqtb7\") pod \"custom-route-timeout-test-kserve-754dbcc69d-4ss4s\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.042391 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.042363    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"]
Sep 17 18:12:15.047027 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.047007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.049463 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.049443    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-p7n99\""
Sep 17 18:12:15.057451 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.057425    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"]
Sep 17 18:12:15.123301 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.123202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ca9d33-cf30-4b23-b414-0df031f2687c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.123301 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.123249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7zh2\" (UniqueName: \"kubernetes.io/projected/51ca9d33-cf30-4b23-b414-0df031f2687c-kube-api-access-d7zh2\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.139174 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.139142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:15.224569 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.224536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ca9d33-cf30-4b23-b414-0df031f2687c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.224569 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.224578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7zh2\" (UniqueName: \"kubernetes.io/projected/51ca9d33-cf30-4b23-b414-0df031f2687c-kube-api-access-d7zh2\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.227465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.227429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ca9d33-cf30-4b23-b414-0df031f2687c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.232873 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.232851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7zh2\" (UniqueName: \"kubernetes.io/projected/51ca9d33-cf30-4b23-b414-0df031f2687c-kube-api-access-d7zh2\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.360493 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.360457    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:15.422038 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.421631    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"]
Sep 17 18:12:15.423268 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:12:15.423236    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590f9ca5_c3bc_417d_a574_2bb66d41ba7e.slice/crio-4ab83cd6f1fadb959b1d534ea654f4a8c176f0f08f1520b30a5c201cecac6e9d WatchSource:0}: Error finding container 4ab83cd6f1fadb959b1d534ea654f4a8c176f0f08f1520b30a5c201cecac6e9d: Status 404 returned error can't find the container with id 4ab83cd6f1fadb959b1d534ea654f4a8c176f0f08f1520b30a5c201cecac6e9d
Sep 17 18:12:15.631992 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:15.631960    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"]
Sep 17 18:12:15.633461 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:12:15.633439    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ca9d33_cf30_4b23_b414_0df031f2687c.slice/crio-543bc32619311f550a96e37da98d06b1566ebaf0cdebbd677012fa2aa625f98c WatchSource:0}: Error finding container 543bc32619311f550a96e37da98d06b1566ebaf0cdebbd677012fa2aa625f98c: Status 404 returned error can't find the container with id 543bc32619311f550a96e37da98d06b1566ebaf0cdebbd677012fa2aa625f98c
Sep 17 18:12:16.176946 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:16.176893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" event={"ID":"590f9ca5-c3bc-417d-a574-2bb66d41ba7e","Type":"ContainerStarted","Data":"d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d"}
Sep 17 18:12:16.177412 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:16.176972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" event={"ID":"590f9ca5-c3bc-417d-a574-2bb66d41ba7e","Type":"ContainerStarted","Data":"4ab83cd6f1fadb959b1d534ea654f4a8c176f0f08f1520b30a5c201cecac6e9d"}
Sep 17 18:12:16.178603 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:16.178572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" event={"ID":"51ca9d33-cf30-4b23-b414-0df031f2687c","Type":"ContainerStarted","Data":"919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7"}
Sep 17 18:12:16.178725 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:16.178621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" event={"ID":"51ca9d33-cf30-4b23-b414-0df031f2687c","Type":"ContainerStarted","Data":"543bc32619311f550a96e37da98d06b1566ebaf0cdebbd677012fa2aa625f98c"}
Sep 17 18:12:16.178725 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:16.178652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:16.215074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:16.215000    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" podStartSLOduration=1.214977993 podStartE2EDuration="1.214977993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:16.214075804 +0000 UTC m=+2296.073667161" watchObservedRunningTime="2026-09-17 18:12:16.214977993 +0000 UTC m=+2296.074569351"
Sep 17 18:12:18.190171 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:18.190138    2566 generic.go:356] "Generic (PLEG): container finished" podID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerID="d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d" exitCode=0
Sep 17 18:12:18.190558 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:18.190214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" event={"ID":"590f9ca5-c3bc-417d-a574-2bb66d41ba7e","Type":"ContainerDied","Data":"d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d"}
Sep 17 18:12:19.197291 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:19.197256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" event={"ID":"590f9ca5-c3bc-417d-a574-2bb66d41ba7e","Type":"ContainerStarted","Data":"0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b"}
Sep 17 18:12:19.219742 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:19.219679    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podStartSLOduration=2.9614682329999997 podStartE2EDuration="5.219658964s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.258190731s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:19.217837368 +0000 UTC m=+2299.077428714" watchObservedRunningTime="2026-09-17 18:12:19.219658964 +0000 UTC m=+2299.079250321"
Sep 17 18:12:20.139328 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:20.139299    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:25.139426 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:25.139385    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:12:25.140789 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:25.140762    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:12:35.139985 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:35.139938    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:12:45.139691 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:45.139598    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:12:47.186507 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:47.186475    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:12:55.140367 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:12:55.140319    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:13:05.139737 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:13:05.139689    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:13:15.139839 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:13:15.139796    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:13:25.140594 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:13:25.140547    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:13:35.140646 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:13:35.140590    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:13:45.139849 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:13:45.139795    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:13:55.139708 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:13:55.139658    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:14:02.587889 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:02.587855    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:14:02.593269 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:02.593245    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:14:05.139605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:05.139549    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:14:15.150060 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:15.149969    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:14:15.158069 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:15.158048    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:14:20.459318 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:20.459271    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"]
Sep 17 18:14:20.464230 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:20.464198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"]
Sep 17 18:14:27.186227 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:27.186186    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:35.482091 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:35.482046    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main" containerID="cri-o://919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7" gracePeriod=45
Sep 17 18:14:35.488116 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:35.488085    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" containerID="cri-o://0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b" gracePeriod=45
Sep 17 18:14:36.150832 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.150766    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:14:36.253419 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.253386    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:14:36.417644 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417593    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-dshm\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.417860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417668    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-model-cache\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.417860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqtb7\" (UniqueName: \"kubernetes.io/projected/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kube-api-access-jqtb7\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.417860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417716    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-cabundle-cert\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.417860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tmp-dir\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.417860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tls-certs\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.417860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-home\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.417860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.417796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kserve-provision-location\") pod \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\" (UID: \"590f9ca5-c3bc-417d-a574-2bb66d41ba7e\") "
Sep 17 18:14:36.418291 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.418125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:36.418680 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.418641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-home" (OuterVolumeSpecName: "home") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:36.418680 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.418669    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:36.420244 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.420212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kube-api-access-jqtb7" (OuterVolumeSpecName: "kube-api-access-jqtb7") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "kube-api-access-jqtb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:36.420382 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.420361    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-dshm" (OuterVolumeSpecName: "dshm") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:36.420454 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.420430    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:36.436720 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.436662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:36.447697 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.447658    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "590f9ca5-c3bc-417d-a574-2bb66d41ba7e" (UID: "590f9ca5-c3bc-417d-a574-2bb66d41ba7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:36.518845 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518819    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqtb7\" (UniqueName: \"kubernetes.io/projected/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kube-api-access-jqtb7\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.518845 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518842    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.519232 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518852    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.519232 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518861    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.519232 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518868    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.519232 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518876    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.519232 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518884    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.519232 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.518892    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590f9ca5-c3bc-417d-a574-2bb66d41ba7e-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.829305 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.829268    2566 generic.go:356] "Generic (PLEG): container finished" podID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerID="0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b" exitCode=0
Sep 17 18:14:36.829474 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.829314    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"
Sep 17 18:14:36.829474 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.829353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" event={"ID":"590f9ca5-c3bc-417d-a574-2bb66d41ba7e","Type":"ContainerDied","Data":"0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b"}
Sep 17 18:14:36.829474 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.829410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s" event={"ID":"590f9ca5-c3bc-417d-a574-2bb66d41ba7e","Type":"ContainerDied","Data":"4ab83cd6f1fadb959b1d534ea654f4a8c176f0f08f1520b30a5c201cecac6e9d"}
Sep 17 18:14:36.829474 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.829433    2566 scope.go:117] "RemoveContainer" containerID="0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b"
Sep 17 18:14:36.838212 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.838129    2566 scope.go:117] "RemoveContainer" containerID="d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d"
Sep 17 18:14:36.848576 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.848554    2566 scope.go:117] "RemoveContainer" containerID="0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b"
Sep 17 18:14:36.848862 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:14:36.848841    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b\": container with ID starting with 0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b not found: ID does not exist" containerID="0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b"
Sep 17 18:14:36.848950 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.848875    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b"} err="failed to get container status \"0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b\": rpc error: code = NotFound desc = could not find container \"0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b\": container with ID starting with 0fae0f0ad7081795c0984d325501c9721850c072c3860776192447dd2d324d7b not found: ID does not exist"
Sep 17 18:14:36.848950 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.848901    2566 scope.go:117] "RemoveContainer" containerID="d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d"
Sep 17 18:14:36.849229 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:14:36.849211    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d\": container with ID starting with d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d not found: ID does not exist" containerID="d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d"
Sep 17 18:14:36.849271 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.849240    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d"} err="failed to get container status \"d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d\": rpc error: code = NotFound desc = could not find container \"d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d\": container with ID starting with d03e123534aa688272671d2ce9aef4c141a50e7143eb808f0ad8dab0bc25935d not found: ID does not exist"
Sep 17 18:14:36.858144 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.858118    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"]
Sep 17 18:14:36.862541 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:36.862515    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-754dbcc69d-4ss4s"]
Sep 17 18:14:37.186447 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:37.186360    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:38.506953 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:38.506899    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" path="/var/lib/kubelet/pods/590f9ca5-c3bc-417d-a574-2bb66d41ba7e/volumes"
Sep 17 18:14:47.186501 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:47.186462    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:47.186897 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:47.186575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:14:57.185988 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:14:57.185949    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:05.734974 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.734944    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:15:05.782123 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.782026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ca9d33-cf30-4b23-b414-0df031f2687c-tls-certs\") pod \"51ca9d33-cf30-4b23-b414-0df031f2687c\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") "
Sep 17 18:15:05.782123 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.782096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7zh2\" (UniqueName: \"kubernetes.io/projected/51ca9d33-cf30-4b23-b414-0df031f2687c-kube-api-access-d7zh2\") pod \"51ca9d33-cf30-4b23-b414-0df031f2687c\" (UID: \"51ca9d33-cf30-4b23-b414-0df031f2687c\") "
Sep 17 18:15:05.784273 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.784246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ca9d33-cf30-4b23-b414-0df031f2687c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51ca9d33-cf30-4b23-b414-0df031f2687c" (UID: "51ca9d33-cf30-4b23-b414-0df031f2687c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:05.784397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.784295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ca9d33-cf30-4b23-b414-0df031f2687c-kube-api-access-d7zh2" (OuterVolumeSpecName: "kube-api-access-d7zh2") pod "51ca9d33-cf30-4b23-b414-0df031f2687c" (UID: "51ca9d33-cf30-4b23-b414-0df031f2687c"). InnerVolumeSpecName "kube-api-access-d7zh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:05.882933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.882874    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ca9d33-cf30-4b23-b414-0df031f2687c-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:15:05.882933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.882902    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7zh2\" (UniqueName: \"kubernetes.io/projected/51ca9d33-cf30-4b23-b414-0df031f2687c-kube-api-access-d7zh2\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:15:05.953341 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.953304    2566 generic.go:356] "Generic (PLEG): container finished" podID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerID="919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7" exitCode=0
Sep 17 18:15:05.953531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.953347    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"
Sep 17 18:15:05.953531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.953394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" event={"ID":"51ca9d33-cf30-4b23-b414-0df031f2687c","Type":"ContainerDied","Data":"919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7"}
Sep 17 18:15:05.953531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.953453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b" event={"ID":"51ca9d33-cf30-4b23-b414-0df031f2687c","Type":"ContainerDied","Data":"543bc32619311f550a96e37da98d06b1566ebaf0cdebbd677012fa2aa625f98c"}
Sep 17 18:15:05.953531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.953483    2566 scope.go:117] "RemoveContainer" containerID="919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7"
Sep 17 18:15:05.962551 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.962533    2566 scope.go:117] "RemoveContainer" containerID="919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7"
Sep 17 18:15:05.962817 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:15:05.962798    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7\": container with ID starting with 919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7 not found: ID does not exist" containerID="919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7"
Sep 17 18:15:05.962870 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.962828    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7"} err="failed to get container status \"919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7\": rpc error: code = NotFound desc = could not find container \"919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7\": container with ID starting with 919a3a2b2b0e804712a27610f64e8e051a0ee9b135e940040b681a4639db8ac7 not found: ID does not exist"
Sep 17 18:15:05.976386 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.976355    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"]
Sep 17 18:15:05.977850 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:05.977824    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587sq2b"]
Sep 17 18:15:06.506109 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:06.506075    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" path="/var/lib/kubelet/pods/51ca9d33-cf30-4b23-b414-0df031f2687c/volumes"
Sep 17 18:15:10.789284 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.789248    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"]
Sep 17 18:15:10.790371 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.790344    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main"
Sep 17 18:15:10.790532 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.790520    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main"
Sep 17 18:15:10.790652 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.790642    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="storage-initializer"
Sep 17 18:15:10.790741 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.790732    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="storage-initializer"
Sep 17 18:15:10.790825 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.790815    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main"
Sep 17 18:15:10.790896 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.790887    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main"
Sep 17 18:15:10.791189 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.791174    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ca9d33-cf30-4b23-b414-0df031f2687c" containerName="main"
Sep 17 18:15:10.791293 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.791283    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="590f9ca5-c3bc-417d-a574-2bb66d41ba7e" containerName="main"
Sep 17 18:15:10.796851 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.796827    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.800214 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.800188    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:15:10.800375 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.800189    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 18:15:10.800375 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.800193    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 18:15:10.800375 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.800272    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-g44lk\""
Sep 17 18:15:10.804755 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.804727    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"]
Sep 17 18:15:10.926181 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54tl\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-kube-api-access-h54tl\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-credential-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926358    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-certs\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-envoy\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926575 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-data\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926575 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-token\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926575 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:10.926575 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:10.926541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-podinfo\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.027788 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.027746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.027961 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.027804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-certs\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.027961 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.027830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-envoy\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.027961 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.027871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-data\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.027961 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.027900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-token\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.027986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-podinfo\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54tl\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-kube-api-access-h54tl\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-credential-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028482 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-certs\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028541 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-credential-socket\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028779 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.028938 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.028818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-data\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.030300 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.030269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-envoy\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.030559 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.030542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-podinfo\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.036673 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.036617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-token\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.037050 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.037026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54tl\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-kube-api-access-h54tl\") pod \"router-gateway-1-openshift-default-544949f8b8-bx5rn\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.109709 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.109668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:11.426405 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.423753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"]
Sep 17 18:15:11.427442 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:15:11.427414    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe5b410_9a0f_48b3_96c2_d296d0f6fb12.slice/crio-0bdd30226ea30a12628fc9da0b920269a4515b15fcd8df2c5bdea174ba475cd4 WatchSource:0}: Error finding container 0bdd30226ea30a12628fc9da0b920269a4515b15fcd8df2c5bdea174ba475cd4: Status 404 returned error can't find the container with id 0bdd30226ea30a12628fc9da0b920269a4515b15fcd8df2c5bdea174ba475cd4
Sep 17 18:15:11.429600 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.429584    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:15:11.979162 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:11.979120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" event={"ID":"abe5b410-9a0f-48b3-96c2-d296d0f6fb12","Type":"ContainerStarted","Data":"0bdd30226ea30a12628fc9da0b920269a4515b15fcd8df2c5bdea174ba475cd4"}
Sep 17 18:15:14.763041 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:14.763001    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:15:14.763353 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:14.763079    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:15:14.763353 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:14.763107    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:15:14.995255 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:14.995220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" event={"ID":"abe5b410-9a0f-48b3-96c2-d296d0f6fb12","Type":"ContainerStarted","Data":"c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47"}
Sep 17 18:15:15.016114 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:15.015997    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podStartSLOduration=1.682934691 podStartE2EDuration="5.015980136s" totalImagesPullingTime="3.333045445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:15.014544651 +0000 UTC m=+2474.874136027" watchObservedRunningTime="2026-09-17 18:15:15.015980136 +0000 UTC m=+2474.875571492"
Sep 17 18:15:15.110808 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:15.110765    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:15.112529 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:15.112494    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 17 18:15:16.110419 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:16.110375    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 17 18:15:17.110553 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:17.110511    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 17 18:15:18.110794 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:18.110755    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 17 18:15:19.110664 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:19.110618    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 17 18:15:20.110824 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:20.110779    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 17 18:15:21.114819 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:21.114787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:21.115221 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:21.115143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:21.115780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:21.115762    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:15:37.814563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.814529    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"]
Sep 17 18:15:37.817918 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.817884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.820647 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.820586    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 18:15:37.820647 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.820625    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-ts8kf\""
Sep 17 18:15:37.820859 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.820664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:15:37.820859 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.820664    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 18:15:37.829240 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.829212    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"]
Sep 17 18:15:37.871627 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-home\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.871816 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kserve-provision-location\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.871816 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tls-certs\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.871816 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14985ef0-cd8c-4184-a710-c0f5ca6f532c-cabundle-cert\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.871816 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-model-cache\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.871985 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45mz\" (UniqueName: \"kubernetes.io/projected/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kube-api-access-w45mz\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.871985 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tmp-dir\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.871985 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.871872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-dshm\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972570 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45mz\" (UniqueName: \"kubernetes.io/projected/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kube-api-access-w45mz\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972570 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tmp-dir\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-dshm\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-home\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kserve-provision-location\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tls-certs\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14985ef0-cd8c-4184-a710-c0f5ca6f532c-cabundle-cert\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.972848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.972796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-model-cache\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.973170 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.973087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tmp-dir\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.973170 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.973147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-home\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.973281 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.973223    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kserve-provision-location\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.973360 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.973329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-model-cache\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.973574 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.973554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14985ef0-cd8c-4184-a710-c0f5ca6f532c-cabundle-cert\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.975135 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.975107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-dshm\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.975531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.975513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tls-certs\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:37.982172 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:37.982150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45mz\" (UniqueName: \"kubernetes.io/projected/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kube-api-access-w45mz\") pod \"router-with-refs-test-kserve-795d5874fd-wh6pj\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:38.130822 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:38.130722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:38.410615 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:38.410589    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"]
Sep 17 18:15:38.412310 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:15:38.412283    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14985ef0_cd8c_4184_a710_c0f5ca6f532c.slice/crio-8938caff47c7356bcd31750f8cc04cad760d8fbe0ecc2feaa445390aef7d20cc WatchSource:0}: Error finding container 8938caff47c7356bcd31750f8cc04cad760d8fbe0ecc2feaa445390aef7d20cc: Status 404 returned error can't find the container with id 8938caff47c7356bcd31750f8cc04cad760d8fbe0ecc2feaa445390aef7d20cc
Sep 17 18:15:39.103397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:39.103350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" event={"ID":"14985ef0-cd8c-4184-a710-c0f5ca6f532c","Type":"ContainerStarted","Data":"3e7d165cfbb35bf567b37d4707b36f27610985becb489c7cce30848a54a6d4ad"}
Sep 17 18:15:39.103872 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:39.103441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" event={"ID":"14985ef0-cd8c-4184-a710-c0f5ca6f532c","Type":"ContainerStarted","Data":"8938caff47c7356bcd31750f8cc04cad760d8fbe0ecc2feaa445390aef7d20cc"}
Sep 17 18:15:41.113328 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:41.113236    2566 generic.go:356] "Generic (PLEG): container finished" podID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerID="3e7d165cfbb35bf567b37d4707b36f27610985becb489c7cce30848a54a6d4ad" exitCode=0
Sep 17 18:15:41.113691 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:41.113312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" event={"ID":"14985ef0-cd8c-4184-a710-c0f5ca6f532c","Type":"ContainerDied","Data":"3e7d165cfbb35bf567b37d4707b36f27610985becb489c7cce30848a54a6d4ad"}
Sep 17 18:15:42.121837 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:42.121790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" event={"ID":"14985ef0-cd8c-4184-a710-c0f5ca6f532c","Type":"ContainerStarted","Data":"e81e72120a8241ebc2d19b9e8324ecc955f39eca490b9b867b8474338673c985"}
Sep 17 18:15:42.131120 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:42.131097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:42.144905 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:42.144852    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podStartSLOduration=2.7694180360000002 podStartE2EDuration="5.14483494s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.375416904s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:42.142617565 +0000 UTC m=+2502.002208921" watchObservedRunningTime="2026-09-17 18:15:42.14483494 +0000 UTC m=+2502.004426295"
Sep 17 18:15:48.131437 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:48.131348    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:15:48.133055 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:48.133022    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:15:58.131610 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:15:58.131554    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:16:08.131768 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:16:08.131717    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:16:18.131391 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:16:18.131346    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:16:28.131305 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:16:28.131262    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:16:38.131852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:16:38.131803    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:16:48.131806 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:16:48.131755    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:16:58.131248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:16:58.131196    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:17:08.131736 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:08.131683    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:17:18.131976 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:18.131853    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:17:28.131307 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:28.131259    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:17:38.141754 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:38.141723    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:17:38.149559 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:38.149532    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:17:42.994288 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:42.994255    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"]
Sep 17 18:17:58.023334 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.023270    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" containerID="cri-o://e81e72120a8241ebc2d19b9e8324ecc955f39eca490b9b867b8474338673c985" gracePeriod=45
Sep 17 18:17:58.142516 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.142471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:17:58.742418 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.742383    2566 generic.go:356] "Generic (PLEG): container finished" podID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerID="e81e72120a8241ebc2d19b9e8324ecc955f39eca490b9b867b8474338673c985" exitCode=0
Sep 17 18:17:58.742561 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.742454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" event={"ID":"14985ef0-cd8c-4184-a710-c0f5ca6f532c","Type":"ContainerDied","Data":"e81e72120a8241ebc2d19b9e8324ecc955f39eca490b9b867b8474338673c985"}
Sep 17 18:17:58.771574 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.771547    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:17:58.863462 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-dshm\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.863462 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tmp-dir\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.863462 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w45mz\" (UniqueName: \"kubernetes.io/projected/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kube-api-access-w45mz\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.863753 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kserve-provision-location\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.863753 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tls-certs\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.863753 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863562    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-home\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.863753 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14985ef0-cd8c-4184-a710-c0f5ca6f532c-cabundle-cert\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.863753 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.863642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-model-cache\") pod \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\" (UID: \"14985ef0-cd8c-4184-a710-c0f5ca6f532c\") "
Sep 17 18:17:58.864103 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.864075    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14985ef0-cd8c-4184-a710-c0f5ca6f532c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:58.864168 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.864100    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-model-cache" (OuterVolumeSpecName: "model-cache") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:58.864511 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.864485    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-home" (OuterVolumeSpecName: "home") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:58.866407 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.866373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-dshm" (OuterVolumeSpecName: "dshm") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:58.866528 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.866431    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:58.866528 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.866442    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kube-api-access-w45mz" (OuterVolumeSpecName: "kube-api-access-w45mz") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "kube-api-access-w45mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:58.876485 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.876457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:58.887597 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.887566    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14985ef0-cd8c-4184-a710-c0f5ca6f532c" (UID: "14985ef0-cd8c-4184-a710-c0f5ca6f532c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:58.964667 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964630    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:58.964667 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964658    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:58.964667 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964667    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:58.964667 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964675    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45mz\" (UniqueName: \"kubernetes.io/projected/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kube-api-access-w45mz\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:58.964984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964690    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:58.964984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964699    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14985ef0-cd8c-4184-a710-c0f5ca6f532c-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:58.964984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964709    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14985ef0-cd8c-4184-a710-c0f5ca6f532c-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:58.964984 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:58.964716    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14985ef0-cd8c-4184-a710-c0f5ca6f532c-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:17:59.748377 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:59.748343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"
Sep 17 18:17:59.748847 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:59.748336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj" event={"ID":"14985ef0-cd8c-4184-a710-c0f5ca6f532c","Type":"ContainerDied","Data":"8938caff47c7356bcd31750f8cc04cad760d8fbe0ecc2feaa445390aef7d20cc"}
Sep 17 18:17:59.748847 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:59.748493    2566 scope.go:117] "RemoveContainer" containerID="e81e72120a8241ebc2d19b9e8324ecc955f39eca490b9b867b8474338673c985"
Sep 17 18:17:59.757905 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:59.757888    2566 scope.go:117] "RemoveContainer" containerID="3e7d165cfbb35bf567b37d4707b36f27610985becb489c7cce30848a54a6d4ad"
Sep 17 18:17:59.771361 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:59.771329    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"]
Sep 17 18:17:59.775097 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:17:59.775072    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-795d5874fd-wh6pj"]
Sep 17 18:18:00.506018 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:00.505983    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" path="/var/lib/kubelet/pods/14985ef0-cd8c-4184-a710-c0f5ca6f532c/volumes"
Sep 17 18:18:25.822983 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.822934    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"]
Sep 17 18:18:25.823469 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.823453    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main"
Sep 17 18:18:25.823514 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.823472    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main"
Sep 17 18:18:25.823514 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.823504    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="storage-initializer"
Sep 17 18:18:25.823592 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.823513    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="storage-initializer"
Sep 17 18:18:25.823626 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.823604    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14985ef0-cd8c-4184-a710-c0f5ca6f532c" containerName="main"
Sep 17 18:18:25.828653 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.828630    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:25.830985 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.830958    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:25.831302 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.831283    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:18:25.831689 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.831666    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:18:25.831826 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.831801    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:25.831943 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.831924    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-gpz8m\""
Sep 17 18:18:25.832075 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.832060    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:18:25.837946 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:25.837922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"]
Sep 17 18:18:26.003304 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-model-cache\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.003304 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-home\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.003563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwph\" (UniqueName: \"kubernetes.io/projected/3474a231-4fe2-4760-a582-b46249c9419c-kube-api-access-zdwph\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.003563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-tmp-dir\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.003563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474a231-4fe2-4760-a582-b46249c9419c-tls-certs\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.003563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-dshm\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.003563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3474a231-4fe2-4760-a582-b46249c9419c-cabundle-cert\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.003563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.003562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105033 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.104906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105033 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.104972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-model-cache\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105033 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.104999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-home\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105033 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwph\" (UniqueName: \"kubernetes.io/projected/3474a231-4fe2-4760-a582-b46249c9419c-kube-api-access-zdwph\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-tmp-dir\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474a231-4fe2-4760-a582-b46249c9419c-tls-certs\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-dshm\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3474a231-4fe2-4760-a582-b46249c9419c-cabundle-cert\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105619 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-model-cache\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105619 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-tmp-dir\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105733 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-home\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.105889 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.105856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3474a231-4fe2-4760-a582-b46249c9419c-cabundle-cert\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.107477 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.107459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-dshm\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.107598 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.107581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474a231-4fe2-4760-a582-b46249c9419c-tls-certs\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.113986 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.113959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwph\" (UniqueName: \"kubernetes.io/projected/3474a231-4fe2-4760-a582-b46249c9419c-kube-api-access-zdwph\") pod \"stop-feature-test-kserve-5cff6864f6-kg2h7\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.140710 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.140670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:26.419365 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.419173    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"]
Sep 17 18:18:26.422097 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:18:26.422071    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3474a231_4fe2_4760_a582_b46249c9419c.slice/crio-7b948dd53cb5214ddecf31854d2d3dc68cc986004e62ea630766fe92430ff705 WatchSource:0}: Error finding container 7b948dd53cb5214ddecf31854d2d3dc68cc986004e62ea630766fe92430ff705: Status 404 returned error can't find the container with id 7b948dd53cb5214ddecf31854d2d3dc68cc986004e62ea630766fe92430ff705
Sep 17 18:18:26.872016 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.871975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" event={"ID":"3474a231-4fe2-4760-a582-b46249c9419c","Type":"ContainerStarted","Data":"8e0d58ccbde999b9ea743441a7b988b863306f858b05d46ea9c7d45fb9d0cd47"}
Sep 17 18:18:26.872518 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:26.872060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" event={"ID":"3474a231-4fe2-4760-a582-b46249c9419c","Type":"ContainerStarted","Data":"7b948dd53cb5214ddecf31854d2d3dc68cc986004e62ea630766fe92430ff705"}
Sep 17 18:18:28.882169 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:28.882135    2566 generic.go:356] "Generic (PLEG): container finished" podID="3474a231-4fe2-4760-a582-b46249c9419c" containerID="8e0d58ccbde999b9ea743441a7b988b863306f858b05d46ea9c7d45fb9d0cd47" exitCode=0
Sep 17 18:18:28.882497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:28.882201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" event={"ID":"3474a231-4fe2-4760-a582-b46249c9419c","Type":"ContainerDied","Data":"8e0d58ccbde999b9ea743441a7b988b863306f858b05d46ea9c7d45fb9d0cd47"}
Sep 17 18:18:29.888763 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:29.888723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" event={"ID":"3474a231-4fe2-4760-a582-b46249c9419c","Type":"ContainerStarted","Data":"7dead3ab79bee93df2a70792e5606277bde389ac5376cbb8c7fc08e76698b5b6"}
Sep 17 18:18:29.912791 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:29.912738    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podStartSLOduration=2.551086077 podStartE2EDuration="4.912718106s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.361632029s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:29.910950343 +0000 UTC m=+2669.770541700" watchObservedRunningTime="2026-09-17 18:18:29.912718106 +0000 UTC m=+2669.772309462"
Sep 17 18:18:30.141576 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:30.141475    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:36.140858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:36.140821    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:18:36.142330 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:36.142301    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:18:38.053990 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:38.053942    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"]
Sep 17 18:18:38.054468 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:38.054211    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy" containerID="cri-o://c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47" gracePeriod=30
Sep 17 18:18:39.797647 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.797612    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"]
Sep 17 18:18:39.800303 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.800279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:39.802989 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.802958    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 18:18:39.803964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.803937    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 18:18:39.804109 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.803979    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-whslj\""
Sep 17 18:18:39.804289 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.804263    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 18:18:39.804442 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.804420    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-gfk5w\""
Sep 17 18:18:39.812164 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.812133    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"]
Sep 17 18:18:39.825079 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.825038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:39.825359 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.825336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnj6\" (UniqueName: \"kubernetes.io/projected/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-kube-api-access-8cnj6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:39.926116 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.926073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:39.926325 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.926135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnj6\" (UniqueName: \"kubernetes.io/projected/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-kube-api-access-8cnj6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:39.928807 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.928784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:39.935634 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:39.935607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnj6\" (UniqueName: \"kubernetes.io/projected/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-kube-api-access-8cnj6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:40.112627 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:40.112586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:40.387102 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:40.387067    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"]
Sep 17 18:18:40.389081 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:18:40.389056    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8120ff2_d98d_41f9_9f19_44c9a44bfacc.slice/crio-66082101b6fd3eee7753be74cbbe48fe0621b6f8bc0d93f1b4e0d4330e32fab5 WatchSource:0}: Error finding container 66082101b6fd3eee7753be74cbbe48fe0621b6f8bc0d93f1b4e0d4330e32fab5: Status 404 returned error can't find the container with id 66082101b6fd3eee7753be74cbbe48fe0621b6f8bc0d93f1b4e0d4330e32fab5
Sep 17 18:18:40.939324 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:40.939288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" event={"ID":"d8120ff2-d98d-41f9-9f19-44c9a44bfacc","Type":"ContainerStarted","Data":"6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032"}
Sep 17 18:18:40.939700 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:40.939344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" event={"ID":"d8120ff2-d98d-41f9-9f19-44c9a44bfacc","Type":"ContainerStarted","Data":"66082101b6fd3eee7753be74cbbe48fe0621b6f8bc0d93f1b4e0d4330e32fab5"}
Sep 17 18:18:40.939700 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:40.939399    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:18:40.955890 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:40.955838    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" podStartSLOduration=1.955817947 podStartE2EDuration="1.955817947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:40.955277505 +0000 UTC m=+2680.814868862" watchObservedRunningTime="2026-09-17 18:18:40.955817947 +0000 UTC m=+2680.815409306"
Sep 17 18:18:43.312427 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.312392    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:18:43.360962 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.360856    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-podinfo\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.360962 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.360901    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-token\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.361229 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.360973    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-socket\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.361229 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.361017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h54tl\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-kube-api-access-h54tl\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.361229 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.361070    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-data\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.361229 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.361096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-certs\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.361229 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.361138    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-envoy\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.361229 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.361187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istiod-ca-cert\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.361576 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.361246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-credential-socket\") pod \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\" (UID: \"abe5b410-9a0f-48b3-96c2-d296d0f6fb12\") "
Sep 17 18:18:43.362029 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.361862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-data" (OuterVolumeSpecName: "istio-data") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:43.362788 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.362151    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:43.362788 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.362417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:43.362788 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.362650    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:43.363242 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.363208    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:43.364246 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.364214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:18:43.364600 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.364572    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-token" (OuterVolumeSpecName: "istio-token") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:43.365082 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.365041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:43.365760 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.365725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-kube-api-access-h54tl" (OuterVolumeSpecName: "kube-api-access-h54tl") pod "abe5b410-9a0f-48b3-96c2-d296d0f6fb12" (UID: "abe5b410-9a0f-48b3-96c2-d296d0f6fb12"). InnerVolumeSpecName "kube-api-access-h54tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:43.462750 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462702    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h54tl\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-kube-api-access-h54tl\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.462750 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462749    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-data\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.463008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462766    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.463008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462778    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-envoy\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.463008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462792    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istiod-ca-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.463008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462804    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-credential-socket\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.463008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462818    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-podinfo\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.463008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462830    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-istio-token\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.463008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.462842    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe5b410-9a0f-48b3-96c2-d296d0f6fb12-workload-socket\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.960036 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.959996    2566 generic.go:356] "Generic (PLEG): container finished" podID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerID="c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47" exitCode=0
Sep 17 18:18:43.960248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.960061    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"
Sep 17 18:18:43.960248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.960093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" event={"ID":"abe5b410-9a0f-48b3-96c2-d296d0f6fb12","Type":"ContainerDied","Data":"c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47"}
Sep 17 18:18:43.960248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.960166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn" event={"ID":"abe5b410-9a0f-48b3-96c2-d296d0f6fb12","Type":"ContainerDied","Data":"0bdd30226ea30a12628fc9da0b920269a4515b15fcd8df2c5bdea174ba475cd4"}
Sep 17 18:18:43.960248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.960196    2566 scope.go:117] "RemoveContainer" containerID="c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47"
Sep 17 18:18:43.969549 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.969530    2566 scope.go:117] "RemoveContainer" containerID="c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47"
Sep 17 18:18:43.969871 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:18:43.969851    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47\": container with ID starting with c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47 not found: ID does not exist" containerID="c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47"
Sep 17 18:18:43.969968 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.969884    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47"} err="failed to get container status \"c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47\": rpc error: code = NotFound desc = could not find container \"c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47\": container with ID starting with c831863f430cca4f6d4d2ef1254d361c67ea0ff5bb98db7c0ea605b706cd6b47 not found: ID does not exist"
Sep 17 18:18:43.985025 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.984996    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"]
Sep 17 18:18:43.989008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:43.988978    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-544949f8b8-bx5rn"]
Sep 17 18:18:44.506964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:44.506933    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" path="/var/lib/kubelet/pods/abe5b410-9a0f-48b3-96c2-d296d0f6fb12/volumes"
Sep 17 18:18:46.141714 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:46.141661    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:18:56.141949 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:18:56.141872    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:19:02.618419 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:02.618384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:19:02.626163 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:02.626143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:19:06.141662 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:06.141618    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:19:11.951543 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:11.951512    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:19:16.141895 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:16.141842    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:19:26.141468 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:26.141426    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:19:36.141289 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:36.141225    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:19:46.141434 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:46.141377    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:19:56.141585 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:19:56.141522    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:20:06.141279 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:06.141229    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:20:16.142158 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:16.142049    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:20:26.151805 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:26.151770    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:20:26.159476 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:26.159448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:20:27.826288 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:27.826249    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"]
Sep 17 18:20:42.848683 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:42.848636    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" containerID="cri-o://7dead3ab79bee93df2a70792e5606277bde389ac5376cbb8c7fc08e76698b5b6" gracePeriod=45
Sep 17 18:20:43.153325 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.153232    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 18:20:43.494895 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.494859    2566 generic.go:356] "Generic (PLEG): container finished" podID="3474a231-4fe2-4760-a582-b46249c9419c" containerID="7dead3ab79bee93df2a70792e5606277bde389ac5376cbb8c7fc08e76698b5b6" exitCode=0
Sep 17 18:20:43.495064 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.494944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" event={"ID":"3474a231-4fe2-4760-a582-b46249c9419c","Type":"ContainerDied","Data":"7dead3ab79bee93df2a70792e5606277bde389ac5376cbb8c7fc08e76698b5b6"}
Sep 17 18:20:43.605515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.605492    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:20:43.652437 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3474a231-4fe2-4760-a582-b46249c9419c-cabundle-cert\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652453    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdwph\" (UniqueName: \"kubernetes.io/projected/3474a231-4fe2-4760-a582-b46249c9419c-kube-api-access-zdwph\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-kserve-provision-location\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652501    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-tmp-dir\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652545    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-model-cache\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652573    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-home\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652878 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-dshm\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652878 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474a231-4fe2-4760-a582-b46249c9419c-tls-certs\") pod \"3474a231-4fe2-4760-a582-b46249c9419c\" (UID: \"3474a231-4fe2-4760-a582-b46249c9419c\") "
Sep 17 18:20:43.652878 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652806    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:43.653056 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652955    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:43.653056 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.652984    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3474a231-4fe2-4760-a582-b46249c9419c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:43.653378 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.653353    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-home" (OuterVolumeSpecName: "home") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:43.655121 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.655090    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3474a231-4fe2-4760-a582-b46249c9419c-kube-api-access-zdwph" (OuterVolumeSpecName: "kube-api-access-zdwph") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "kube-api-access-zdwph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:43.655294 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.655270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3474a231-4fe2-4760-a582-b46249c9419c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:43.655389 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.655368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-dshm" (OuterVolumeSpecName: "dshm") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:43.665905 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.665866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:43.677998 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.677977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3474a231-4fe2-4760-a582-b46249c9419c" (UID: "3474a231-4fe2-4760-a582-b46249c9419c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:43.753964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.753879    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdwph\" (UniqueName: \"kubernetes.io/projected/3474a231-4fe2-4760-a582-b46249c9419c-kube-api-access-zdwph\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:43.753964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.753906    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:43.753964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.753934    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:43.753964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.753943    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:43.753964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.753951    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3474a231-4fe2-4760-a582-b46249c9419c-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:43.753964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.753959    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474a231-4fe2-4760-a582-b46249c9419c-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:43.753964 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:43.753967    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3474a231-4fe2-4760-a582-b46249c9419c-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:20:44.501219 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:44.501189    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"
Sep 17 18:20:44.505655 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:44.505627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7" event={"ID":"3474a231-4fe2-4760-a582-b46249c9419c","Type":"ContainerDied","Data":"7b948dd53cb5214ddecf31854d2d3dc68cc986004e62ea630766fe92430ff705"}
Sep 17 18:20:44.505802 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:44.505673    2566 scope.go:117] "RemoveContainer" containerID="7dead3ab79bee93df2a70792e5606277bde389ac5376cbb8c7fc08e76698b5b6"
Sep 17 18:20:44.515145 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:44.515127    2566 scope.go:117] "RemoveContainer" containerID="8e0d58ccbde999b9ea743441a7b988b863306f858b05d46ea9c7d45fb9d0cd47"
Sep 17 18:20:44.526774 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:44.526745    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"]
Sep 17 18:20:44.532634 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:44.532612    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-kg2h7"]
Sep 17 18:20:46.506837 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:46.506801    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3474a231-4fe2-4760-a582-b46249c9419c" path="/var/lib/kubelet/pods/3474a231-4fe2-4760-a582-b46249c9419c/volumes"
Sep 17 18:20:53.824229 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824195    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"]
Sep 17 18:20:53.824699 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824684    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="storage-initializer"
Sep 17 18:20:53.824748 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824703    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="storage-initializer"
Sep 17 18:20:53.824748 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824728    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main"
Sep 17 18:20:53.824748 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824737    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main"
Sep 17 18:20:53.824845 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824757    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy"
Sep 17 18:20:53.824845 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824767    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy"
Sep 17 18:20:53.824845 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824839    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3474a231-4fe2-4760-a582-b46249c9419c" containerName="main"
Sep 17 18:20:53.824961 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.824852    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe5b410-9a0f-48b3-96c2-d296d0f6fb12" containerName="istio-proxy"
Sep 17 18:20:53.829864 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.829843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.832653 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.832625    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 18:20:53.832653 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.832643    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-gpz8m\""
Sep 17 18:20:53.832898 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.832719    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:20:53.832898 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.832631    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:20:53.833022 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.832978    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:20:53.833086 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.833069    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:20:53.837224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.837198    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"]
Sep 17 18:20:53.937985 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.937949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.938195 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.938010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4fg4\" (UniqueName: \"kubernetes.io/projected/17747b81-f9b3-47ff-82d5-5bf855f97d08-kube-api-access-j4fg4\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.938195 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.938093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-tmp-dir\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.938195 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.938138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-dshm\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.938195 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.938181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-home\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.938392 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.938221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17747b81-f9b3-47ff-82d5-5bf855f97d08-tls-certs\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.938392 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.938248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17747b81-f9b3-47ff-82d5-5bf855f97d08-cabundle-cert\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:53.938392 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:53.938281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-model-cache\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039070 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039238 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4fg4\" (UniqueName: \"kubernetes.io/projected/17747b81-f9b3-47ff-82d5-5bf855f97d08-kube-api-access-j4fg4\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039238 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-tmp-dir\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039238 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-dshm\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039238 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-home\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039479 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17747b81-f9b3-47ff-82d5-5bf855f97d08-tls-certs\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039479 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17747b81-f9b3-47ff-82d5-5bf855f97d08-cabundle-cert\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039479 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-model-cache\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039641 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039641 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-tmp-dir\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-home\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.039749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-model-cache\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.040010 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.039991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17747b81-f9b3-47ff-82d5-5bf855f97d08-cabundle-cert\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.041537 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.041512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-dshm\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.041781 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.041766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17747b81-f9b3-47ff-82d5-5bf855f97d08-tls-certs\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.047848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.047822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4fg4\" (UniqueName: \"kubernetes.io/projected/17747b81-f9b3-47ff-82d5-5bf855f97d08-kube-api-access-j4fg4\") pod \"stop-feature-test-kserve-5cff6864f6-4w595\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.142273 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.142194    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:20:54.415741 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.415713    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"]
Sep 17 18:20:54.417254 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:20:54.417229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17747b81_f9b3_47ff_82d5_5bf855f97d08.slice/crio-90f4dc1e1b2b86e2c6c5a49205ffe0d5150a25a288037dbddbc4f22b92f7ebd3 WatchSource:0}: Error finding container 90f4dc1e1b2b86e2c6c5a49205ffe0d5150a25a288037dbddbc4f22b92f7ebd3: Status 404 returned error can't find the container with id 90f4dc1e1b2b86e2c6c5a49205ffe0d5150a25a288037dbddbc4f22b92f7ebd3
Sep 17 18:20:54.541273 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.541228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" event={"ID":"17747b81-f9b3-47ff-82d5-5bf855f97d08","Type":"ContainerStarted","Data":"10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d"}
Sep 17 18:20:54.541454 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:54.541290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" event={"ID":"17747b81-f9b3-47ff-82d5-5bf855f97d08","Type":"ContainerStarted","Data":"90f4dc1e1b2b86e2c6c5a49205ffe0d5150a25a288037dbddbc4f22b92f7ebd3"}
Sep 17 18:20:57.556236 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:57.556202    2566 generic.go:356] "Generic (PLEG): container finished" podID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerID="10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d" exitCode=0
Sep 17 18:20:57.556784 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:57.556261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" event={"ID":"17747b81-f9b3-47ff-82d5-5bf855f97d08","Type":"ContainerDied","Data":"10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d"}
Sep 17 18:20:58.562365 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:58.562329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" event={"ID":"17747b81-f9b3-47ff-82d5-5bf855f97d08","Type":"ContainerStarted","Data":"b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a"}
Sep 17 18:20:58.586423 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:58.586363    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podStartSLOduration=3.279338977 podStartE2EDuration="5.586342483s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.307003506s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:58.584306554 +0000 UTC m=+2818.443897912" watchObservedRunningTime="2026-09-17 18:20:58.586342483 +0000 UTC m=+2818.445933840"
Sep 17 18:20:59.143007 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:20:59.142966    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:21:04.143132 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:04.143096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:21:04.144864 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:04.144834    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:21:14.142816 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:14.142772    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:21:24.143665 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:24.143616    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:21:29.060527 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:29.060490    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"]
Sep 17 18:21:31.951982 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:31.951934    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:34.142840 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:34.142799    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:21:41.951439 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:41.951405    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:44.082232 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:44.082135    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main" containerID="cri-o://6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032" gracePeriod=45
Sep 17 18:21:44.143456 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:44.143404    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:21:51.952228 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:51.952189    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:51.952616 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:51.952303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:21:54.142702 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:21:54.142653    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:22:01.952731 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:01.952691    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:04.142791 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:04.142745    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:22:11.951905 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:11.951861    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:14.143059 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.143010    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:22:14.344859 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.344794    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:22:14.478285 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.478248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-tls-certs\") pod \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") "
Sep 17 18:22:14.478478 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.478368    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cnj6\" (UniqueName: \"kubernetes.io/projected/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-kube-api-access-8cnj6\") pod \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\" (UID: \"d8120ff2-d98d-41f9-9f19-44c9a44bfacc\") "
Sep 17 18:22:14.480601 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.480572    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-kube-api-access-8cnj6" (OuterVolumeSpecName: "kube-api-access-8cnj6") pod "d8120ff2-d98d-41f9-9f19-44c9a44bfacc" (UID: "d8120ff2-d98d-41f9-9f19-44c9a44bfacc"). InnerVolumeSpecName "kube-api-access-8cnj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:14.480713 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.480671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8120ff2-d98d-41f9-9f19-44c9a44bfacc" (UID: "d8120ff2-d98d-41f9-9f19-44c9a44bfacc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:14.579438 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.579403    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnj6\" (UniqueName: \"kubernetes.io/projected/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-kube-api-access-8cnj6\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:22:14.579438 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.579431    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8120ff2-d98d-41f9-9f19-44c9a44bfacc-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:22:14.905259 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.905224    2566 generic.go:356] "Generic (PLEG): container finished" podID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerID="6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032" exitCode=0
Sep 17 18:22:14.905439 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.905272    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"
Sep 17 18:22:14.905439 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.905299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" event={"ID":"d8120ff2-d98d-41f9-9f19-44c9a44bfacc","Type":"ContainerDied","Data":"6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032"}
Sep 17 18:22:14.905439 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.905344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc" event={"ID":"d8120ff2-d98d-41f9-9f19-44c9a44bfacc","Type":"ContainerDied","Data":"66082101b6fd3eee7753be74cbbe48fe0621b6f8bc0d93f1b4e0d4330e32fab5"}
Sep 17 18:22:14.905439 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.905365    2566 scope.go:117] "RemoveContainer" containerID="6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032"
Sep 17 18:22:14.913942 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.913919    2566 scope.go:117] "RemoveContainer" containerID="6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032"
Sep 17 18:22:14.914266 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:22:14.914244    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032\": container with ID starting with 6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032 not found: ID does not exist" containerID="6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032"
Sep 17 18:22:14.914374 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.914271    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032"} err="failed to get container status \"6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032\": rpc error: code = NotFound desc = could not find container \"6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032\": container with ID starting with 6f0c1a62fcb537a6f8490a5ed4d35483c7220d9fb449d4caf70b1ad962b2f032 not found: ID does not exist"
Sep 17 18:22:14.923959 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.923934    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"]
Sep 17 18:22:14.928683 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:14.928657    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedrvjc"]
Sep 17 18:22:16.506797 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:16.506765    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" path="/var/lib/kubelet/pods/d8120ff2-d98d-41f9-9f19-44c9a44bfacc/volumes"
Sep 17 18:22:23.646150 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.646114    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"]
Sep 17 18:22:23.647129 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.647103    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main"
Sep 17 18:22:23.647265 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.647158    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main"
Sep 17 18:22:23.647326 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.647267    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8120ff2-d98d-41f9-9f19-44c9a44bfacc" containerName="main"
Sep 17 18:22:23.652016 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.651991    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:23.655821 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.655797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zpnsk\""
Sep 17 18:22:23.656008 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.655983    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:22:23.656199 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.655866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:22:23.656316 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.655942    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:22:23.656400 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.655811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-kxd48\""
Sep 17 18:22:23.660806 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.660779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"]
Sep 17 18:22:23.761564 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.761518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c938555-99fc-480e-bccd-460eccfe0041-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:23.761746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.761597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cld6d\" (UniqueName: \"kubernetes.io/projected/5c938555-99fc-480e-bccd-460eccfe0041-kube-api-access-cld6d\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:23.862717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.862682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c938555-99fc-480e-bccd-460eccfe0041-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:23.862717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.862720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cld6d\" (UniqueName: \"kubernetes.io/projected/5c938555-99fc-480e-bccd-460eccfe0041-kube-api-access-cld6d\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:23.865320 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.865298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c938555-99fc-480e-bccd-460eccfe0041-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:23.873048 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.873017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cld6d\" (UniqueName: \"kubernetes.io/projected/5c938555-99fc-480e-bccd-460eccfe0041-kube-api-access-cld6d\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:23.967717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:23.967619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:24.143464 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:24.143418    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:22:24.256867 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:24.256838    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"]
Sep 17 18:22:24.259716 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:22:24.259682    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c938555_99fc_480e_bccd_460eccfe0041.slice/crio-57697ac942d50999b7aefc8b27b3a72720844f6717611bcbb7e402e1fb062e5a WatchSource:0}: Error finding container 57697ac942d50999b7aefc8b27b3a72720844f6717611bcbb7e402e1fb062e5a: Status 404 returned error can't find the container with id 57697ac942d50999b7aefc8b27b3a72720844f6717611bcbb7e402e1fb062e5a
Sep 17 18:22:24.952942 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:24.952892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" event={"ID":"5c938555-99fc-480e-bccd-460eccfe0041","Type":"ContainerStarted","Data":"4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a"}
Sep 17 18:22:24.953327 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:24.952976    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:22:24.953327 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:24.952996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" event={"ID":"5c938555-99fc-480e-bccd-460eccfe0041","Type":"ContainerStarted","Data":"57697ac942d50999b7aefc8b27b3a72720844f6717611bcbb7e402e1fb062e5a"}
Sep 17 18:22:24.970653 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:24.970604    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" podStartSLOduration=1.9705857020000002 podStartE2EDuration="1.970585702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:24.96917251 +0000 UTC m=+2904.828763872" watchObservedRunningTime="2026-09-17 18:22:24.970585702 +0000 UTC m=+2904.830177057"
Sep 17 18:22:34.143415 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:34.143369    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:22:44.143272 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:44.143233    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:22:54.153038 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:54.152994    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:22:54.161087 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:54.161052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:22:55.777073 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:55.777031    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"]
Sep 17 18:22:55.960040 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:22:55.960013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:23:11.116763 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.116703    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main" containerID="cri-o://b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a" gracePeriod=45
Sep 17 18:23:11.862777 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.862751    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:23:11.995072 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.994986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4fg4\" (UniqueName: \"kubernetes.io/projected/17747b81-f9b3-47ff-82d5-5bf855f97d08-kube-api-access-j4fg4\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995072 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995052    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-home\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995298 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17747b81-f9b3-47ff-82d5-5bf855f97d08-cabundle-cert\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995298 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-kserve-provision-location\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995298 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995128    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-model-cache\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995298 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995155    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-tmp-dir\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995298 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17747b81-f9b3-47ff-82d5-5bf855f97d08-tls-certs\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995298 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995242    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-dshm\") pod \"17747b81-f9b3-47ff-82d5-5bf855f97d08\" (UID: \"17747b81-f9b3-47ff-82d5-5bf855f97d08\") "
Sep 17 18:23:11.995624 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-model-cache" (OuterVolumeSpecName: "model-cache") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:11.995624 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17747b81-f9b3-47ff-82d5-5bf855f97d08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:11.995624 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995566    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:11.995861 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.995824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-home" (OuterVolumeSpecName: "home") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:11.997549 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.997521    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-dshm" (OuterVolumeSpecName: "dshm") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:11.997790 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.997758    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17747b81-f9b3-47ff-82d5-5bf855f97d08-kube-api-access-j4fg4" (OuterVolumeSpecName: "kube-api-access-j4fg4") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "kube-api-access-j4fg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:11.997940 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:11.997851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17747b81-f9b3-47ff-82d5-5bf855f97d08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:12.007743 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.007703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:12.019355 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.019331    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17747b81-f9b3-47ff-82d5-5bf855f97d08" (UID: "17747b81-f9b3-47ff-82d5-5bf855f97d08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:12.096861 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.096830    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.096861 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.096859    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17747b81-f9b3-47ff-82d5-5bf855f97d08-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.097048 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.096871    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.097048 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.096880    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.097048 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.096889    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17747b81-f9b3-47ff-82d5-5bf855f97d08-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.097048 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.096896    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17747b81-f9b3-47ff-82d5-5bf855f97d08-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.097048 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.096903    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4fg4\" (UniqueName: \"kubernetes.io/projected/17747b81-f9b3-47ff-82d5-5bf855f97d08-kube-api-access-j4fg4\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.163239 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.163200    2566 generic.go:356] "Generic (PLEG): container finished" podID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerID="b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a" exitCode=0
Sep 17 18:23:12.163641 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.163258    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"
Sep 17 18:23:12.163641 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.163285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" event={"ID":"17747b81-f9b3-47ff-82d5-5bf855f97d08","Type":"ContainerDied","Data":"b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a"}
Sep 17 18:23:12.163641 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.163344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595" event={"ID":"17747b81-f9b3-47ff-82d5-5bf855f97d08","Type":"ContainerDied","Data":"90f4dc1e1b2b86e2c6c5a49205ffe0d5150a25a288037dbddbc4f22b92f7ebd3"}
Sep 17 18:23:12.163641 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.163366    2566 scope.go:117] "RemoveContainer" containerID="b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a"
Sep 17 18:23:12.172126 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.172108    2566 scope.go:117] "RemoveContainer" containerID="10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d"
Sep 17 18:23:12.179740 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.179722    2566 scope.go:117] "RemoveContainer" containerID="b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a"
Sep 17 18:23:12.180034 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:23:12.180016    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a\": container with ID starting with b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a not found: ID does not exist" containerID="b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a"
Sep 17 18:23:12.180089 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.180044    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a"} err="failed to get container status \"b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a\": rpc error: code = NotFound desc = could not find container \"b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a\": container with ID starting with b8375ca15b67789dd37b544535ca0177327174b3842b5cb8065b39a89501285a not found: ID does not exist"
Sep 17 18:23:12.180089 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.180062    2566 scope.go:117] "RemoveContainer" containerID="10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d"
Sep 17 18:23:12.180281 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:23:12.180264    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d\": container with ID starting with 10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d not found: ID does not exist" containerID="10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d"
Sep 17 18:23:12.180334 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.180288    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d"} err="failed to get container status \"10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d\": rpc error: code = NotFound desc = could not find container \"10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d\": container with ID starting with 10ef54409364ce2fcc4fb3b2d7883e537fce963bf1cefefadb0c3f72061ea52d not found: ID does not exist"
Sep 17 18:23:12.188067 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.188046    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"]
Sep 17 18:23:12.197975 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.197953    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff6864f6-4w595"]
Sep 17 18:23:12.506685 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:23:12.506652    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" path="/var/lib/kubelet/pods/17747b81-f9b3-47ff-82d5-5bf855f97d08/volumes"
Sep 17 18:24:02.647064 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:02.647036    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:24:02.656844 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:02.656825    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:24:05.442398 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.442367    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"]
Sep 17 18:24:05.442790 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.442709    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="storage-initializer"
Sep 17 18:24:05.442790 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.442720    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="storage-initializer"
Sep 17 18:24:05.442790 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.442729    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main"
Sep 17 18:24:05.442790 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.442734    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main"
Sep 17 18:24:05.442790 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.442790    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="17747b81-f9b3-47ff-82d5-5bf855f97d08" containerName="main"
Sep 17 18:24:05.446896 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.446879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.449316 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.449288    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:24:05.449316 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.449289    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 18:24:05.449589 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.449563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 18:24:05.449718 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.449668    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 18:24:05.449718 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.449690    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:24:05.449982 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.449949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1905b3c8-467b-487a-992f-9305bfc938a4-tls-certs\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.450097 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntbbd\" (UniqueName: \"kubernetes.io/projected/1905b3c8-467b-487a-992f-9305bfc938a4-kube-api-access-ntbbd\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.450097 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450014    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-8sbzs\""
Sep 17 18:24:05.450097 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-tmp-dir\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.450097 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.450260 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-dshm\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.450260 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-home\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.450260 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-model-cache\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.450260 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.450234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1905b3c8-467b-487a-992f-9305bfc938a4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.457831 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.457811    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"]
Sep 17 18:24:05.551050 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-model-cache\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1905b3c8-467b-487a-992f-9305bfc938a4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1905b3c8-467b-487a-992f-9305bfc938a4-tls-certs\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551387 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntbbd\" (UniqueName: \"kubernetes.io/projected/1905b3c8-467b-487a-992f-9305bfc938a4-kube-api-access-ntbbd\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551459 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-tmp-dir\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551459 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551586 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-dshm\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551586 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-model-cache\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551586 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-home\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551841 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-tmp-dir\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.551933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-home\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.552038 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.551978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1905b3c8-467b-487a-992f-9305bfc938a4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.553717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.553693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-dshm\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.553717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.553710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1905b3c8-467b-487a-992f-9305bfc938a4-tls-certs\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.560230 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.560208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntbbd\" (UniqueName: \"kubernetes.io/projected/1905b3c8-467b-487a-992f-9305bfc938a4-kube-api-access-ntbbd\") pod \"lora-single-adapter-test-kserve-668f5b96c8-pbccx\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:05.758290 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:05.758207    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:06.037614 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:06.037586    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"]
Sep 17 18:24:06.039670 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:24:06.039642    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1905b3c8_467b_487a_992f_9305bfc938a4.slice/crio-bde26b796a2ee894d55698023de04ad9492e7dc8e2b5c4be7d15fe20935e26e5 WatchSource:0}: Error finding container bde26b796a2ee894d55698023de04ad9492e7dc8e2b5c4be7d15fe20935e26e5: Status 404 returned error can't find the container with id bde26b796a2ee894d55698023de04ad9492e7dc8e2b5c4be7d15fe20935e26e5
Sep 17 18:24:06.387429 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:06.387392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" event={"ID":"1905b3c8-467b-487a-992f-9305bfc938a4","Type":"ContainerStarted","Data":"68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b"}
Sep 17 18:24:06.387671 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:06.387456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" event={"ID":"1905b3c8-467b-487a-992f-9305bfc938a4","Type":"ContainerStarted","Data":"bde26b796a2ee894d55698023de04ad9492e7dc8e2b5c4be7d15fe20935e26e5"}
Sep 17 18:24:09.401642 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:09.401606    2566 generic.go:356] "Generic (PLEG): container finished" podID="1905b3c8-467b-487a-992f-9305bfc938a4" containerID="68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b" exitCode=0
Sep 17 18:24:09.402020 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:09.401678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" event={"ID":"1905b3c8-467b-487a-992f-9305bfc938a4","Type":"ContainerDied","Data":"68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b"}
Sep 17 18:24:10.408773 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:10.408737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" event={"ID":"1905b3c8-467b-487a-992f-9305bfc938a4","Type":"ContainerStarted","Data":"a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370"}
Sep 17 18:24:10.434254 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:10.434195    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podStartSLOduration=2.665707995 podStartE2EDuration="5.434179699s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.768471704s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:10.431770041 +0000 UTC m=+3010.291361388" watchObservedRunningTime="2026-09-17 18:24:10.434179699 +0000 UTC m=+3010.293771054"
Sep 17 18:24:10.758418 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:10.758326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:15.758688 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:15.758651    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:24:15.760479 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:15.760450    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:24:25.759149 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:25.759101    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:24:35.759554 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:35.759494    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:24:45.758891 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:45.758800    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:24:55.759628 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:24:55.759586    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:25:05.759002 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:05.758958    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:25:13.097224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:13.097183    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"]
Sep 17 18:25:15.759076 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:15.759026    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:25:15.960144 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:15.960088    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:25:25.768743 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:25.768712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:25:25.776825 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:25.776799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:25:25.959670 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:25.959631    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:25:28.117174 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:28.117137    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main" containerID="cri-o://4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a" gracePeriod=45
Sep 17 18:25:35.959189 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:35.959152    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:25:35.959677 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:35.959254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:25:45.959981 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:45.959940    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:25:51.333021 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:51.332990    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-668f5b96c8-pbccx_1905b3c8-467b-487a-992f-9305bfc938a4/storage-initializer/0.log"
Sep 17 18:25:51.342355 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:51.342331    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-668f5b96c8-pbccx_1905b3c8-467b-487a-992f-9305bfc938a4/main/0.log"
Sep 17 18:25:52.723353 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:52.723316    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"]
Sep 17 18:25:55.959898 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:55.959861    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:25:58.374441 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.374414    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:25:58.456038 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.455948    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c938555-99fc-480e-bccd-460eccfe0041-tls-certs\") pod \"5c938555-99fc-480e-bccd-460eccfe0041\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") "
Sep 17 18:25:58.456038 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.455988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cld6d\" (UniqueName: \"kubernetes.io/projected/5c938555-99fc-480e-bccd-460eccfe0041-kube-api-access-cld6d\") pod \"5c938555-99fc-480e-bccd-460eccfe0041\" (UID: \"5c938555-99fc-480e-bccd-460eccfe0041\") "
Sep 17 18:25:58.458318 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.458287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c938555-99fc-480e-bccd-460eccfe0041-kube-api-access-cld6d" (OuterVolumeSpecName: "kube-api-access-cld6d") pod "5c938555-99fc-480e-bccd-460eccfe0041" (UID: "5c938555-99fc-480e-bccd-460eccfe0041"). InnerVolumeSpecName "kube-api-access-cld6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:58.458433 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.458289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c938555-99fc-480e-bccd-460eccfe0041-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c938555-99fc-480e-bccd-460eccfe0041" (UID: "5c938555-99fc-480e-bccd-460eccfe0041"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:58.556815 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.556775    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c938555-99fc-480e-bccd-460eccfe0041-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:25:58.556815 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.556809    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cld6d\" (UniqueName: \"kubernetes.io/projected/5c938555-99fc-480e-bccd-460eccfe0041-kube-api-access-cld6d\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:25:58.891607 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.891572    2566 generic.go:356] "Generic (PLEG): container finished" podID="5c938555-99fc-480e-bccd-460eccfe0041" containerID="4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a" exitCode=0
Sep 17 18:25:58.891815 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.891637    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"
Sep 17 18:25:58.891815 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.891649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" event={"ID":"5c938555-99fc-480e-bccd-460eccfe0041","Type":"ContainerDied","Data":"4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a"}
Sep 17 18:25:58.891815 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.891716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8" event={"ID":"5c938555-99fc-480e-bccd-460eccfe0041","Type":"ContainerDied","Data":"57697ac942d50999b7aefc8b27b3a72720844f6717611bcbb7e402e1fb062e5a"}
Sep 17 18:25:58.891815 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.891747    2566 scope.go:117] "RemoveContainer" containerID="4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a"
Sep 17 18:25:58.900201 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.900182    2566 scope.go:117] "RemoveContainer" containerID="4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a"
Sep 17 18:25:58.900466 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:25:58.900447    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a\": container with ID starting with 4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a not found: ID does not exist" containerID="4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a"
Sep 17 18:25:58.900511 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.900477    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a"} err="failed to get container status \"4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a\": rpc error: code = NotFound desc = could not find container \"4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a\": container with ID starting with 4028195ffe7e7b57cac3896b8c8d16473e5f5e87584b1b220891a821c85bf83a not found: ID does not exist"
Sep 17 18:25:58.909928 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.909880    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"]
Sep 17 18:25:58.916446 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:25:58.916422    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48x5vw8"]
Sep 17 18:26:00.506039 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:00.506006    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c938555-99fc-480e-bccd-460eccfe0041" path="/var/lib/kubelet/pods/5c938555-99fc-480e-bccd-460eccfe0041/volumes"
Sep 17 18:26:07.745402 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:07.745355    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main" containerID="cri-o://a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370" gracePeriod=45
Sep 17 18:26:08.397960 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.397935    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:26:08.442371 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntbbd\" (UniqueName: \"kubernetes.io/projected/1905b3c8-467b-487a-992f-9305bfc938a4-kube-api-access-ntbbd\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442384    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1905b3c8-467b-487a-992f-9305bfc938a4-tls-certs\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442437    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-home\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442487    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1905b3c8-467b-487a-992f-9305bfc938a4-cabundle-cert\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442542    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-kserve-provision-location\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442796 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442576    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-dshm\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442796 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-model-cache\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442796 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-tmp-dir\") pod \"1905b3c8-467b-487a-992f-9305bfc938a4\" (UID: \"1905b3c8-467b-487a-992f-9305bfc938a4\") "
Sep 17 18:26:08.442965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.442884    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-home" (OuterVolumeSpecName: "home") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:08.443255 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.443227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:08.443360 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.443298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1905b3c8-467b-487a-992f-9305bfc938a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:08.445316 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.445286    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1905b3c8-467b-487a-992f-9305bfc938a4-kube-api-access-ntbbd" (OuterVolumeSpecName: "kube-api-access-ntbbd") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "kube-api-access-ntbbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:08.445434 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.445382    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1905b3c8-467b-487a-992f-9305bfc938a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:08.445492 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.445443    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-dshm" (OuterVolumeSpecName: "dshm") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:08.460506 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.460466    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:08.469838 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.469755    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1905b3c8-467b-487a-992f-9305bfc938a4" (UID: "1905b3c8-467b-487a-992f-9305bfc938a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:08.543776 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543738    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.543776 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543772    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.544000 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543789    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.544000 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543803    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.544000 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543814    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntbbd\" (UniqueName: \"kubernetes.io/projected/1905b3c8-467b-487a-992f-9305bfc938a4-kube-api-access-ntbbd\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.544000 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543828    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1905b3c8-467b-487a-992f-9305bfc938a4-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.544000 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543842    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1905b3c8-467b-487a-992f-9305bfc938a4-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.544000 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.543853    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1905b3c8-467b-487a-992f-9305bfc938a4-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:26:08.937541 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.937489    2566 generic.go:356] "Generic (PLEG): container finished" podID="1905b3c8-467b-487a-992f-9305bfc938a4" containerID="a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370" exitCode=0
Sep 17 18:26:08.938085 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.937546    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"
Sep 17 18:26:08.938085 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.937580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" event={"ID":"1905b3c8-467b-487a-992f-9305bfc938a4","Type":"ContainerDied","Data":"a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370"}
Sep 17 18:26:08.938085 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.937633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx" event={"ID":"1905b3c8-467b-487a-992f-9305bfc938a4","Type":"ContainerDied","Data":"bde26b796a2ee894d55698023de04ad9492e7dc8e2b5c4be7d15fe20935e26e5"}
Sep 17 18:26:08.938085 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.937663    2566 scope.go:117] "RemoveContainer" containerID="a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370"
Sep 17 18:26:08.947080 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.947055    2566 scope.go:117] "RemoveContainer" containerID="68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b"
Sep 17 18:26:08.956084 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.956063    2566 scope.go:117] "RemoveContainer" containerID="a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370"
Sep 17 18:26:08.956403 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:26:08.956378    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370\": container with ID starting with a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370 not found: ID does not exist" containerID="a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370"
Sep 17 18:26:08.956458 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.956413    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370"} err="failed to get container status \"a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370\": rpc error: code = NotFound desc = could not find container \"a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370\": container with ID starting with a4eeb8fceaddcb0646685977b4c74cbe9bcbefadfce53c1a4abde767184e1370 not found: ID does not exist"
Sep 17 18:26:08.956458 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.956431    2566 scope.go:117] "RemoveContainer" containerID="68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b"
Sep 17 18:26:08.956732 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:26:08.956714    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b\": container with ID starting with 68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b not found: ID does not exist" containerID="68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b"
Sep 17 18:26:08.956794 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.956736    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b"} err="failed to get container status \"68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b\": rpc error: code = NotFound desc = could not find container \"68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b\": container with ID starting with 68381322800d7d45651695d840c615e4756a4a0abbc515f386060e9c30168f4b not found: ID does not exist"
Sep 17 18:26:08.963566 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.963541    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"]
Sep 17 18:26:08.966965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:08.966929    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-668f5b96c8-pbccx"]
Sep 17 18:26:10.506524 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:10.506488    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" path="/var/lib/kubelet/pods/1905b3c8-467b-487a-992f-9305bfc938a4/volumes"
Sep 17 18:26:18.169122 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169019    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"]
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169413    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="storage-initializer"
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169427    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="storage-initializer"
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169446    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main"
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169452    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main"
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169464    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main"
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169470    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main"
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169522    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1905b3c8-467b-487a-992f-9305bfc938a4" containerName="main"
Sep 17 18:26:18.169571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.169530    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c938555-99fc-480e-bccd-460eccfe0041" containerName="main"
Sep 17 18:26:18.174455 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.174435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.177233 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.177208    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:26:18.177362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.177279    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 18:26:18.177362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.177335    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-zbb9p\""
Sep 17 18:26:18.177362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.177348    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:26:18.177531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.177371    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-4mggk\""
Sep 17 18:26:18.177531 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.177420    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:26:18.178076 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.178057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:26:18.187362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.187331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"]
Sep 17 18:26:18.328568 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.328764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-dshm\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.328764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-model-cache\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.328764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-home\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.328764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.328764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291a922e-a12f-466d-b800-d3e55d9a7ee5-tls-certs\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.328764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/291a922e-a12f-466d-b800-d3e55d9a7ee5-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.328764 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.328750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47d48\" (UniqueName: \"kubernetes.io/projected/291a922e-a12f-466d-b800-d3e55d9a7ee5-kube-api-access-47d48\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430323 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-model-cache\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430323 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-home\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430323 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430626 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291a922e-a12f-466d-b800-d3e55d9a7ee5-tls-certs\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430626 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/291a922e-a12f-466d-b800-d3e55d9a7ee5-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430626 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47d48\" (UniqueName: \"kubernetes.io/projected/291a922e-a12f-466d-b800-d3e55d9a7ee5-kube-api-access-47d48\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430626 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430626 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-dshm\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430870 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-home\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430870 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.430966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.430938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-model-cache\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.431117 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.431098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/291a922e-a12f-466d-b800-d3e55d9a7ee5-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.432867 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.432847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-dshm\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.433020 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.433005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291a922e-a12f-466d-b800-d3e55d9a7ee5-tls-certs\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.440771 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.440748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47d48\" (UniqueName: \"kubernetes.io/projected/291a922e-a12f-466d-b800-d3e55d9a7ee5-kube-api-access-47d48\") pod \"router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.484210 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.484173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.765511 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.765481    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"]
Sep 17 18:26:18.767237 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:26:18.767213    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291a922e_a12f_466d_b800_d3e55d9a7ee5.slice/crio-eafa74931da98da311593c2a73b6a403a4672123f119eb67b8a4b3dba43ea5b3 WatchSource:0}: Error finding container eafa74931da98da311593c2a73b6a403a4672123f119eb67b8a4b3dba43ea5b3: Status 404 returned error can't find the container with id eafa74931da98da311593c2a73b6a403a4672123f119eb67b8a4b3dba43ea5b3
Sep 17 18:26:18.983339 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.983303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerStarted","Data":"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"}
Sep 17 18:26:18.983530 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.983373    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:18.983530 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:18.983391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerStarted","Data":"eafa74931da98da311593c2a73b6a403a4672123f119eb67b8a4b3dba43ea5b3"}
Sep 17 18:26:19.989506 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:19.989465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerStarted","Data":"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"}
Sep 17 18:26:22.005780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:22.005736    2566 generic.go:356] "Generic (PLEG): container finished" podID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerID="21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c" exitCode=0
Sep 17 18:26:22.006270 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:22.005805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerDied","Data":"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"}
Sep 17 18:26:23.013228 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:23.013182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerStarted","Data":"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"}
Sep 17 18:26:23.041826 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:23.041768    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podStartSLOduration=2.6804948299999998 podStartE2EDuration="5.041753503s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.361258673s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:23.040484777 +0000 UTC m=+3142.900076147" watchObservedRunningTime="2026-09-17 18:26:23.041753503 +0000 UTC m=+3142.901344859"
Sep 17 18:26:28.485308 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:28.485263    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:28.485763 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:28.485360    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:28.486773 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:28.486736    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:26:28.499207 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:28.499177    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:26:38.485045 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:38.484982    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:26:48.485282 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:48.485228    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:26:58.485497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:26:58.485441    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:27:08.485222 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:27:08.485166    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:27:18.485593 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:27:18.485539    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:27:28.485590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:27:28.485541    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:27:38.485516 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:27:38.485462    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:27:48.485022 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:27:48.484893    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:27:58.484644 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:27:58.484583    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:28:08.485432 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:08.485367    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Sep 17 18:28:18.495055 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:18.495017    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:28:18.511173 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:18.511140    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:28:28.166873 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:28.166838    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"]
Sep 17 18:28:43.190298 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:43.190247    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main" containerID="cri-o://3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb" gracePeriod=45
Sep 17 18:28:43.807099 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:43.807042    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="llm-d-routing-sidecar" containerID="cri-o://08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2" gracePeriod=45
Sep 17 18:28:44.053853 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.053821    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:28:44.170490 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170450    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47d48\" (UniqueName: \"kubernetes.io/projected/291a922e-a12f-466d-b800-d3e55d9a7ee5-kube-api-access-47d48\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.170709 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-kserve-provision-location\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.170709 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170563    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-model-cache\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.170709 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170591    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-dshm\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.170709 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170634    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/291a922e-a12f-466d-b800-d3e55d9a7ee5-cabundle-cert\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.170709 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170661    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-home\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.170709 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291a922e-a12f-466d-b800-d3e55d9a7ee5-tls-certs\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.171053 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170740    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-tmp-dir\") pod \"291a922e-a12f-466d-b800-d3e55d9a7ee5\" (UID: \"291a922e-a12f-466d-b800-d3e55d9a7ee5\") "
Sep 17 18:28:44.171053 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.170861    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-model-cache" (OuterVolumeSpecName: "model-cache") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.171171 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.171054    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.171171 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.171107    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291a922e-a12f-466d-b800-d3e55d9a7ee5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:44.171617 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.171584    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-home" (OuterVolumeSpecName: "home") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.173241 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.173202    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291a922e-a12f-466d-b800-d3e55d9a7ee5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:44.173356 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.173299    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291a922e-a12f-466d-b800-d3e55d9a7ee5-kube-api-access-47d48" (OuterVolumeSpecName: "kube-api-access-47d48") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "kube-api-access-47d48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:44.173356 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.173336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-dshm" (OuterVolumeSpecName: "dshm") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.187494 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.187452    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.197742 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.197719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "291a922e-a12f-466d-b800-d3e55d9a7ee5" (UID: "291a922e-a12f-466d-b800-d3e55d9a7ee5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.271633 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.271594    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.271633 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.271629    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.271633 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.271640    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/291a922e-a12f-466d-b800-d3e55d9a7ee5-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.271880 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.271650    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.271880 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.271658    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291a922e-a12f-466d-b800-d3e55d9a7ee5-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.271880 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.271666    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291a922e-a12f-466d-b800-d3e55d9a7ee5-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.271880 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.271673    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47d48\" (UniqueName: \"kubernetes.io/projected/291a922e-a12f-466d-b800-d3e55d9a7ee5-kube-api-access-47d48\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.660216 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.660177    2566 generic.go:356] "Generic (PLEG): container finished" podID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerID="3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb" exitCode=0
Sep 17 18:28:44.660216 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.660203    2566 generic.go:356] "Generic (PLEG): container finished" podID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerID="08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2" exitCode=0
Sep 17 18:28:44.660497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.660238    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"
Sep 17 18:28:44.660497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.660259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerDied","Data":"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"}
Sep 17 18:28:44.660497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.660317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerDied","Data":"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"}
Sep 17 18:28:44.660497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.660335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv" event={"ID":"291a922e-a12f-466d-b800-d3e55d9a7ee5","Type":"ContainerDied","Data":"eafa74931da98da311593c2a73b6a403a4672123f119eb67b8a4b3dba43ea5b3"}
Sep 17 18:28:44.660497 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.660356    2566 scope.go:117] "RemoveContainer" containerID="3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"
Sep 17 18:28:44.668846 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.668822    2566 scope.go:117] "RemoveContainer" containerID="21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"
Sep 17 18:28:44.676472 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.676452    2566 scope.go:117] "RemoveContainer" containerID="08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"
Sep 17 18:28:44.680013 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.679987    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"]
Sep 17 18:28:44.684210 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.684186    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f5fcc74b5-f6qkv"]
Sep 17 18:28:44.685153 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.685131    2566 scope.go:117] "RemoveContainer" containerID="3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"
Sep 17 18:28:44.685486 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:28:44.685467    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb\": container with ID starting with 3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb not found: ID does not exist" containerID="3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"
Sep 17 18:28:44.685534 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.685497    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"} err="failed to get container status \"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb\": rpc error: code = NotFound desc = could not find container \"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb\": container with ID starting with 3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb not found: ID does not exist"
Sep 17 18:28:44.685534 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.685517    2566 scope.go:117] "RemoveContainer" containerID="21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"
Sep 17 18:28:44.685803 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:28:44.685785    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c\": container with ID starting with 21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c not found: ID does not exist" containerID="21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"
Sep 17 18:28:44.685844 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.685809    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"} err="failed to get container status \"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c\": rpc error: code = NotFound desc = could not find container \"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c\": container with ID starting with 21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c not found: ID does not exist"
Sep 17 18:28:44.685844 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.685827    2566 scope.go:117] "RemoveContainer" containerID="08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"
Sep 17 18:28:44.686081 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:28:44.686064    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2\": container with ID starting with 08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2 not found: ID does not exist" containerID="08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"
Sep 17 18:28:44.686147 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.686085    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"} err="failed to get container status \"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2\": rpc error: code = NotFound desc = could not find container \"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2\": container with ID starting with 08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2 not found: ID does not exist"
Sep 17 18:28:44.686147 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.686097    2566 scope.go:117] "RemoveContainer" containerID="3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"
Sep 17 18:28:44.686335 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.686319    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb"} err="failed to get container status \"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb\": rpc error: code = NotFound desc = could not find container \"3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb\": container with ID starting with 3c764de8eb006ce27ccf95545fa628b2ef32b1fcee5c10439222422eb8ea0feb not found: ID does not exist"
Sep 17 18:28:44.686375 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.686337    2566 scope.go:117] "RemoveContainer" containerID="21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"
Sep 17 18:28:44.686569 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.686549    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c"} err="failed to get container status \"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c\": rpc error: code = NotFound desc = could not find container \"21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c\": container with ID starting with 21ea3db19b04cba008e79f67a881005690a44045ad924ba1b1313b6f8211301c not found: ID does not exist"
Sep 17 18:28:44.686647 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.686572    2566 scope.go:117] "RemoveContainer" containerID="08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"
Sep 17 18:28:44.686789 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:44.686771    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2"} err="failed to get container status \"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2\": rpc error: code = NotFound desc = could not find container \"08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2\": container with ID starting with 08e15d15c05b1af068a46f53a80da087ce68c1389a25274c397ff3e7e885dec2 not found: ID does not exist"
Sep 17 18:28:46.506673 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:28:46.506639    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" path="/var/lib/kubelet/pods/291a922e-a12f-466d-b800-d3e55d9a7ee5/volumes"
Sep 17 18:29:02.675865 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:02.675834    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:29:02.687848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:02.687818    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:29:29.382569 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.382526    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"]
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.382877    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main"
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.382888    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main"
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.382899    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="llm-d-routing-sidecar"
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.382905    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="llm-d-routing-sidecar"
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.382949    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="storage-initializer"
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.382957    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="storage-initializer"
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.383010    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="llm-d-routing-sidecar"
Sep 17 18:29:29.383043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.383022    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="291a922e-a12f-466d-b800-d3e55d9a7ee5" containerName="main"
Sep 17 18:29:29.385967 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.385950    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:29.388288 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.388264    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:29:29.388419 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.388265    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-52grw\""
Sep 17 18:29:29.389227 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.389211    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:29:29.389305 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.389212    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:29:29.392552 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.392521    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"]
Sep 17 18:29:29.557688 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.557645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9cgd\" (UniqueName: \"kubernetes.io/projected/969a8dd2-cdb4-4ab8-8e7f-60bc09db0519-kube-api-access-b9cgd\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:29.658442 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.658349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cgd\" (UniqueName: \"kubernetes.io/projected/969a8dd2-cdb4-4ab8-8e7f-60bc09db0519-kube-api-access-b9cgd\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:29.667667 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:29.667636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cgd\" (UniqueName: \"kubernetes.io/projected/969a8dd2-cdb4-4ab8-8e7f-60bc09db0519-kube-api-access-b9cgd\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:31.779759 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:31.779720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:31.785810 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:31.785784    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:31.881064 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:31.881030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:31.881272 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:31.881192    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:31.884277 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:31.884256    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") DevicePath \"csi-35cb7f3e816de5857de991754e6c5ea5aad4f69c336d944e66bfcfd3ea278e89\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:32.052983 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.052949    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c640e3025f003c03c46dae8c816de9b130cbdc1424bf5ecca1cf2d33f4551660/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:32.066885 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.066851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e2e-pvc-lora-download-8mvmp\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:32.096337 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.096299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:32.367772 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.367743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"]
Sep 17 18:29:32.370447 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:29:32.370419    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod969a8dd2_cdb4_4ab8_8e7f_60bc09db0519.slice/crio-847eafc48cd7c7b12c97a10e8e26fb7b5f2ad10ff64912c139980c691fed0c19 WatchSource:0}: Error finding container 847eafc48cd7c7b12c97a10e8e26fb7b5f2ad10ff64912c139980c691fed0c19: Status 404 returned error can't find the container with id 847eafc48cd7c7b12c97a10e8e26fb7b5f2ad10ff64912c139980c691fed0c19
Sep 17 18:29:32.404427 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.404392    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"]
Sep 17 18:29:32.409025 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.409004    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.411425 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.411397    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:29:32.411624 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.411399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 18:29:32.411722 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.411665    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-hh5dl\""
Sep 17 18:29:32.411722 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.411663    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 18:29:32.411722 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.411687    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 18:29:32.411722 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.411705    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 18:29:32.420397 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.420368    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"]
Sep 17 18:29:32.587426 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.587426 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.587657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.587657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d109ca-2d12-480f-a9e9-f30ae0d5a952-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.587657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwpf\" (UniqueName: \"kubernetes.io/projected/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kube-api-access-fjwpf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.587657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.587791 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.587791 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.587769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.688567 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.688567 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.688567 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.688867 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.688867 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d109ca-2d12-480f-a9e9-f30ae0d5a952-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.688867 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwpf\" (UniqueName: \"kubernetes.io/projected/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kube-api-access-fjwpf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.688867 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.689101 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.688887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.689101 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.689004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.689101 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.689046    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.689486 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.689454    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d109ca-2d12-480f-a9e9-f30ae0d5a952-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.689600 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.689523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.689600 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.689527    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.691224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.691199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.691352 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.691337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.697553 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.697524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwpf\" (UniqueName: \"kubernetes.io/projected/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kube-api-access-fjwpf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.731144 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.731109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:32.864452 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.864406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp" event={"ID":"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519","Type":"ContainerStarted","Data":"7d61fdc2e5399ec4832c4e973b18b307d269f75abad629f764b9b7551bb99c71"}
Sep 17 18:29:32.864452 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:32.864457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp" event={"ID":"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519","Type":"ContainerStarted","Data":"847eafc48cd7c7b12c97a10e8e26fb7b5f2ad10ff64912c139980c691fed0c19"}
Sep 17 18:29:33.008811 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:33.008673    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp" podStartSLOduration=7.008654294 podStartE2EDuration="7.008654294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:32.882078619 +0000 UTC m=+3332.741669974" watchObservedRunningTime="2026-09-17 18:29:33.008654294 +0000 UTC m=+3332.868245650"
Sep 17 18:29:33.009419 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:33.009395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"]
Sep 17 18:29:33.011065 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:29:33.011038    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d109ca_2d12_480f_a9e9_f30ae0d5a952.slice/crio-19bcb6dfdfc4302584a9c24ee51c75704b65a36b39b46fde31a225e22ddd1202 WatchSource:0}: Error finding container 19bcb6dfdfc4302584a9c24ee51c75704b65a36b39b46fde31a225e22ddd1202: Status 404 returned error can't find the container with id 19bcb6dfdfc4302584a9c24ee51c75704b65a36b39b46fde31a225e22ddd1202
Sep 17 18:29:33.873067 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:33.873014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" event={"ID":"68d109ca-2d12-480f-a9e9-f30ae0d5a952","Type":"ContainerStarted","Data":"c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b"}
Sep 17 18:29:33.873643 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:33.873103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" event={"ID":"68d109ca-2d12-480f-a9e9-f30ae0d5a952","Type":"ContainerStarted","Data":"19bcb6dfdfc4302584a9c24ee51c75704b65a36b39b46fde31a225e22ddd1202"}
Sep 17 18:29:34.878807 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:34.878774    2566 generic.go:356] "Generic (PLEG): container finished" podID="969a8dd2-cdb4-4ab8-8e7f-60bc09db0519" containerID="7d61fdc2e5399ec4832c4e973b18b307d269f75abad629f764b9b7551bb99c71" exitCode=0
Sep 17 18:29:34.879286 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:34.878847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp" event={"ID":"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519","Type":"ContainerDied","Data":"7d61fdc2e5399ec4832c4e973b18b307d269f75abad629f764b9b7551bb99c71"}
Sep 17 18:29:35.884196 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:35.884160    2566 generic.go:356] "Generic (PLEG): container finished" podID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerID="c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b" exitCode=0
Sep 17 18:29:35.884670 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:35.884242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" event={"ID":"68d109ca-2d12-480f-a9e9-f30ae0d5a952","Type":"ContainerDied","Data":"c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b"}
Sep 17 18:29:36.034188 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.034150    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:36.125335 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.125292    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9cgd\" (UniqueName: \"kubernetes.io/projected/969a8dd2-cdb4-4ab8-8e7f-60bc09db0519-kube-api-access-b9cgd\") pod \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") "
Sep 17 18:29:36.125562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.125521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\" (UID: \"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519\") "
Sep 17 18:29:36.128599 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.128550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969a8dd2-cdb4-4ab8-8e7f-60bc09db0519-kube-api-access-b9cgd" (OuterVolumeSpecName: "kube-api-access-b9cgd") pod "969a8dd2-cdb4-4ab8-8e7f-60bc09db0519" (UID: "969a8dd2-cdb4-4ab8-8e7f-60bc09db0519"). InnerVolumeSpecName "kube-api-access-b9cgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:36.129975 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.129946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf" (OuterVolumeSpecName: "model-storage") pod "969a8dd2-cdb4-4ab8-8e7f-60bc09db0519" (UID: "969a8dd2-cdb4-4ab8-8e7f-60bc09db0519"). InnerVolumeSpecName "pvc-45be254d-0821-4e52-8cb1-7e118558d2b4". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:29:36.227039 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.226931    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") on node \"ip-10-0-135-84.ec2.internal\" "
Sep 17 18:29:36.227039 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.226990    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9cgd\" (UniqueName: \"kubernetes.io/projected/969a8dd2-cdb4-4ab8-8e7f-60bc09db0519-kube-api-access-b9cgd\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:29:36.891173 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.891127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" event={"ID":"68d109ca-2d12-480f-a9e9-f30ae0d5a952","Type":"ContainerStarted","Data":"adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f"}
Sep 17 18:29:36.892505 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.892477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp" event={"ID":"969a8dd2-cdb4-4ab8-8e7f-60bc09db0519","Type":"ContainerDied","Data":"847eafc48cd7c7b12c97a10e8e26fb7b5f2ad10ff64912c139980c691fed0c19"}
Sep 17 18:29:36.892631 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.892511    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"
Sep 17 18:29:36.892631 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.892515    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="847eafc48cd7c7b12c97a10e8e26fb7b5f2ad10ff64912c139980c691fed0c19"
Sep 17 18:29:36.919795 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:36.919739    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podStartSLOduration=2.370224989 podStartE2EDuration="4.919721292s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.549496303s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:36.916761509 +0000 UTC m=+3336.776352885" watchObservedRunningTime="2026-09-17 18:29:36.919721292 +0000 UTC m=+3336.779312648"
Sep 17 18:29:37.203616 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:37.203582    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-45be254d-0821-4e52-8cb1-7e118558d2b4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf") on node "ip-10-0-135-84.ec2.internal"
Sep 17 18:29:37.237021 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:37.236977    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"csi-35cb7f3e816de5857de991754e6c5ea5aad4f69c336d944e66bfcfd3ea278e89\""
Sep 17 18:29:37.732272 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:37.732228    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:42.731932 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:42.731858    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:29:42.733354 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:42.733324    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:29:52.732481 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:29:52.732436    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:30:02.731876 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:30:02.731831    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:30:12.732376 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:30:12.732329    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:30:22.732030 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:30:22.731979    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:30:32.732421 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:30:32.732370    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:30:42.732265 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:30:42.732215    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:30:52.732157 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:30:52.732061    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:31:02.731717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:02.731663    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:31:12.732538 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:12.732482    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:31:22.732035 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:22.731987    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 18:31:32.741357 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:32.741320    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:31:32.749285 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:32.749245    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:31:37.680552 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:37.680516    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"]
Sep 17 18:31:52.703811 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:52.703765    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main" containerID="cri-o://adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f" gracePeriod=45
Sep 17 18:31:53.466435 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.466404    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:31:53.484657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484620    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjwpf\" (UniqueName: \"kubernetes.io/projected/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kube-api-access-fjwpf\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.484852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tmp-dir\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.484852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d109ca-2d12-480f-a9e9-f30ae0d5a952-cabundle-cert\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.484852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484728    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tls-certs\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.484852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-home\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.484852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kserve-provision-location\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.484852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484825    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-dshm\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.484852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.484852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-model-cache\") pod \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\" (UID: \"68d109ca-2d12-480f-a9e9-f30ae0d5a952\") "
Sep 17 18:31:53.485269 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.485162    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d109ca-2d12-480f-a9e9-f30ae0d5a952-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:53.485345 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.485320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-model-cache" (OuterVolumeSpecName: "model-cache") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:53.485590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.485566    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-home" (OuterVolumeSpecName: "home") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:53.487401 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.487373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kube-api-access-fjwpf" (OuterVolumeSpecName: "kube-api-access-fjwpf") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "kube-api-access-fjwpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:53.487515 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.487424    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:53.487697 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.487662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-dshm" (OuterVolumeSpecName: "dshm") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:53.503467 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.503421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:53.511623 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.511586    2566 generic.go:356] "Generic (PLEG): container finished" podID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerID="adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f" exitCode=0
Sep 17 18:31:53.511800 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.511633    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"
Sep 17 18:31:53.511800 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.511660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" event={"ID":"68d109ca-2d12-480f-a9e9-f30ae0d5a952","Type":"ContainerDied","Data":"adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f"}
Sep 17 18:31:53.511800 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.511726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg" event={"ID":"68d109ca-2d12-480f-a9e9-f30ae0d5a952","Type":"ContainerDied","Data":"19bcb6dfdfc4302584a9c24ee51c75704b65a36b39b46fde31a225e22ddd1202"}
Sep 17 18:31:53.511800 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.511756    2566 scope.go:117] "RemoveContainer" containerID="adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f"
Sep 17 18:31:53.515005 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.514977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68d109ca-2d12-480f-a9e9-f30ae0d5a952" (UID: "68d109ca-2d12-480f-a9e9-f30ae0d5a952"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:53.522462 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.522443    2566 scope.go:117] "RemoveContainer" containerID="c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b"
Sep 17 18:31:53.530567 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.530539    2566 scope.go:117] "RemoveContainer" containerID="adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f"
Sep 17 18:31:53.530896 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:31:53.530879    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f\": container with ID starting with adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f not found: ID does not exist" containerID="adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f"
Sep 17 18:31:53.530974 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.530922    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f"} err="failed to get container status \"adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f\": rpc error: code = NotFound desc = could not find container \"adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f\": container with ID starting with adbc3fc7b8f0b54cc070d00becf532a52e4307bcad303969d35970c45d58136f not found: ID does not exist"
Sep 17 18:31:53.530974 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.530946    2566 scope.go:117] "RemoveContainer" containerID="c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b"
Sep 17 18:31:53.531217 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:31:53.531194    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b\": container with ID starting with c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b not found: ID does not exist" containerID="c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b"
Sep 17 18:31:53.531268 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.531226    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b"} err="failed to get container status \"c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b\": rpc error: code = NotFound desc = could not find container \"c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b\": container with ID starting with c3a284753150fbd2ff84d0d0edd23c4b62acf0a47447301eb68e702bef4bb66b not found: ID does not exist"
Sep 17 18:31:53.585590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585547    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d109ca-2d12-480f-a9e9-f30ae0d5a952-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.585590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585583    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.585590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585592    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.585590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585602    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.585871 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585611    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.585871 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585624    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.585871 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585634    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjwpf\" (UniqueName: \"kubernetes.io/projected/68d109ca-2d12-480f-a9e9-f30ae0d5a952-kube-api-access-fjwpf\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.585871 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.585642    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d109ca-2d12-480f-a9e9-f30ae0d5a952-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:31:53.836936 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.836878    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"]
Sep 17 18:31:53.841283 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:53.841250    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-754dbb6554gnkwg"]
Sep 17 18:31:54.506712 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:31:54.506672    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" path="/var/lib/kubelet/pods/68d109ca-2d12-480f-a9e9-f30ae0d5a952/volumes"
Sep 17 18:32:06.802859 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.802821    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"]
Sep 17 18:32:06.803299 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803282    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969a8dd2-cdb4-4ab8-8e7f-60bc09db0519" containerName="storage-initializer"
Sep 17 18:32:06.803299 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803299    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="969a8dd2-cdb4-4ab8-8e7f-60bc09db0519" containerName="storage-initializer"
Sep 17 18:32:06.803393 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803311    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="storage-initializer"
Sep 17 18:32:06.803393 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803317    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="storage-initializer"
Sep 17 18:32:06.803393 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803331    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main"
Sep 17 18:32:06.803393 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803337    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main"
Sep 17 18:32:06.803393 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803393    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="969a8dd2-cdb4-4ab8-8e7f-60bc09db0519" containerName="storage-initializer"
Sep 17 18:32:06.803584 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.803409    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d109ca-2d12-480f-a9e9-f30ae0d5a952" containerName="main"
Sep 17 18:32:06.806462 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.806435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:06.810205 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.810181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-p55qs\""
Sep 17 18:32:06.810205 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.810199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 18:32:06.810402 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.810227    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 18:32:06.810402 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.810181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-29xph\""
Sep 17 18:32:06.810402 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.810181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 18:32:06.820484 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.820453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"]
Sep 17 18:32:06.827291 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.827262    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:32:06.831039 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.831013    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.833439 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.833411    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:32:06.833601 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.833491    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 18:32:06.833685 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.833630    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-cwqqz\""
Sep 17 18:32:06.844616 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.844584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:32:06.896022 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.895975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88cj\" (UniqueName: \"kubernetes.io/projected/bfb93956-b13e-4470-8a61-979f9c6a5b07-kube-api-access-x88cj\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.896204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75324d8-a2eb-4581-b9c7-93b4636446c0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:06.896204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfb93956-b13e-4470-8a61-979f9c6a5b07-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.896204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.896204 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb93956-b13e-4470-8a61-979f9c6a5b07-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.896390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f256z\" (UniqueName: \"kubernetes.io/projected/a75324d8-a2eb-4581-b9c7-93b4636446c0-kube-api-access-f256z\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:06.896390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.896390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.896390 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.896525 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.896406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.996866 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.996823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997086 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.996874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997086 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.996946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997086 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.996989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88cj\" (UniqueName: \"kubernetes.io/projected/bfb93956-b13e-4470-8a61-979f9c6a5b07-kube-api-access-x88cj\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997086 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75324d8-a2eb-4581-b9c7-93b4636446c0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:06.997086 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfb93956-b13e-4470-8a61-979f9c6a5b07-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997086 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997408 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb93956-b13e-4470-8a61-979f9c6a5b07-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997408 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f256z\" (UniqueName: \"kubernetes.io/projected/a75324d8-a2eb-4581-b9c7-93b4636446c0-kube-api-access-f256z\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:06.997408 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997408 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997665 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997730 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997787 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.997847 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.997833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfb93956-b13e-4470-8a61-979f9c6a5b07-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.999409 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.999383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.999835 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.999811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb93956-b13e-4470-8a61-979f9c6a5b07-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:06.999957 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:06.999942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75324d8-a2eb-4581-b9c7-93b4636446c0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:07.005744 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.005715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88cj\" (UniqueName: \"kubernetes.io/projected/bfb93956-b13e-4470-8a61-979f9c6a5b07-kube-api-access-x88cj\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:07.006027 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.006009    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f256z\" (UniqueName: \"kubernetes.io/projected/a75324d8-a2eb-4581-b9c7-93b4636446c0-kube-api-access-f256z\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:07.117890 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.117795    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:07.142176 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.142141    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:07.397670 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.397631    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"]
Sep 17 18:32:07.400299 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:32:07.400273    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75324d8_a2eb_4581_b9c7_93b4636446c0.slice/crio-12f7974227d5f6305eeb548d68fd3e911fe5ece5b839333b158f4730aef45ca0 WatchSource:0}: Error finding container 12f7974227d5f6305eeb548d68fd3e911fe5ece5b839333b158f4730aef45ca0: Status 404 returned error can't find the container with id 12f7974227d5f6305eeb548d68fd3e911fe5ece5b839333b158f4730aef45ca0
Sep 17 18:32:07.430087 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.429950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:32:07.433083 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:32:07.433056    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb93956_b13e_4470_8a61_979f9c6a5b07.slice/crio-7f5a3095c4bfe6b1f3af162f05f7b05ae2effe8ab76d7e9a9ac113535c54bb71 WatchSource:0}: Error finding container 7f5a3095c4bfe6b1f3af162f05f7b05ae2effe8ab76d7e9a9ac113535c54bb71: Status 404 returned error can't find the container with id 7f5a3095c4bfe6b1f3af162f05f7b05ae2effe8ab76d7e9a9ac113535c54bb71
Sep 17 18:32:07.573200 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.573142    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" event={"ID":"a75324d8-a2eb-4581-b9c7-93b4636446c0","Type":"ContainerStarted","Data":"0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469"}
Sep 17 18:32:07.573398 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.573209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" event={"ID":"a75324d8-a2eb-4581-b9c7-93b4636446c0","Type":"ContainerStarted","Data":"12f7974227d5f6305eeb548d68fd3e911fe5ece5b839333b158f4730aef45ca0"}
Sep 17 18:32:07.573398 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.573285    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:07.575015 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.574978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bfb93956-b13e-4470-8a61-979f9c6a5b07","Type":"ContainerStarted","Data":"ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5"}
Sep 17 18:32:07.575157 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.575030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bfb93956-b13e-4470-8a61-979f9c6a5b07","Type":"ContainerStarted","Data":"7f5a3095c4bfe6b1f3af162f05f7b05ae2effe8ab76d7e9a9ac113535c54bb71"}
Sep 17 18:32:07.597852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:07.597792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" podStartSLOduration=1.5977736120000001 podStartE2EDuration="1.597773612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:07.596857237 +0000 UTC m=+3487.456448595" watchObservedRunningTime="2026-09-17 18:32:07.597773612 +0000 UTC m=+3487.457364969"
Sep 17 18:32:10.594227 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:10.594191    2566 generic.go:356] "Generic (PLEG): container finished" podID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerID="ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5" exitCode=0
Sep 17 18:32:10.594666 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:10.594262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bfb93956-b13e-4470-8a61-979f9c6a5b07","Type":"ContainerDied","Data":"ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5"}
Sep 17 18:32:11.604375 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:11.604331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bfb93956-b13e-4470-8a61-979f9c6a5b07","Type":"ContainerStarted","Data":"678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50"}
Sep 17 18:32:11.626862 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:11.626801    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.184255464 podStartE2EDuration="5.626784752s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.442529288s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:11.624324699 +0000 UTC m=+3491.483916055" watchObservedRunningTime="2026-09-17 18:32:11.626784752 +0000 UTC m=+3491.486376109"
Sep 17 18:32:12.143406 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:12.143365    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:17.142865 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:17.142761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:32:17.144367 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:17.144330    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:32:20.401675 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:20.401633    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"]
Sep 17 18:32:20.408374 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:20.408346    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-8mvmp"]
Sep 17 18:32:20.506578 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:20.506544    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969a8dd2-cdb4-4ab8-8e7f-60bc09db0519" path="/var/lib/kubelet/pods/969a8dd2-cdb4-4ab8-8e7f-60bc09db0519/volumes"
Sep 17 18:32:27.143146 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:27.143097    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:32:37.143324 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:37.143276    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:32:38.582336 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:38.582304    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:32:47.143448 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:47.143402    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:32:57.143829 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:32:57.143777    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:33:07.142904 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:33:07.142861    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:33:17.143873 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:33:17.143818    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:33:27.143682 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:33:27.143622    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:33:37.143535 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:33:37.143466    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:33:47.143147 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:33:47.143054    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:33:57.153231 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:33:57.153195    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:33:57.161332 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:33:57.161303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:34:02.707555 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:02.707526    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:34:02.721894 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:02.721865    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:34:03.600224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:03.600188    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"]
Sep 17 18:34:03.611638 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:03.611605    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:34:08.581807 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:08.581766    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:18.581547 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:18.581504    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:18.621248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:18.621208    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main" containerID="cri-o://0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469" gracePeriod=45
Sep 17 18:34:18.634343 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:18.634303    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" containerID="cri-o://678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50" gracePeriod=45
Sep 17 18:34:19.154146 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.154105    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 18:34:19.398976 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.398942    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:34:19.518461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518354    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-home\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfb93956-b13e-4470-8a61-979f9c6a5b07-cabundle-cert\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518432    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb93956-b13e-4470-8a61-979f9c6a5b07-tls-certs\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518461 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-model-cache\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518832 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x88cj\" (UniqueName: \"kubernetes.io/projected/bfb93956-b13e-4470-8a61-979f9c6a5b07-kube-api-access-x88cj\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518832 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-kserve-provision-location\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518832 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518541    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-tmp-dir\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518832 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-dshm\") pod \"bfb93956-b13e-4470-8a61-979f9c6a5b07\" (UID: \"bfb93956-b13e-4470-8a61-979f9c6a5b07\") "
Sep 17 18:34:19.518832 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.518780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-model-cache" (OuterVolumeSpecName: "model-cache") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:19.519196 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.519171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-home" (OuterVolumeSpecName: "home") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:19.519386 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.519218    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb93956-b13e-4470-8a61-979f9c6a5b07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:34:19.521058 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.521031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb93956-b13e-4470-8a61-979f9c6a5b07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:34:19.521281 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.521246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb93956-b13e-4470-8a61-979f9c6a5b07-kube-api-access-x88cj" (OuterVolumeSpecName: "kube-api-access-x88cj") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "kube-api-access-x88cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:19.521375 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.521301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-dshm" (OuterVolumeSpecName: "dshm") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:19.532017 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.531983    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:19.544129 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.544066    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bfb93956-b13e-4470-8a61-979f9c6a5b07" (UID: "bfb93956-b13e-4470-8a61-979f9c6a5b07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:19.620123 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620082    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x88cj\" (UniqueName: \"kubernetes.io/projected/bfb93956-b13e-4470-8a61-979f9c6a5b07-kube-api-access-x88cj\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:19.620123 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620117    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:19.620123 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620128    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:19.620629 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620137    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:19.620629 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620145    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:19.620629 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620153    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfb93956-b13e-4470-8a61-979f9c6a5b07-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:19.620629 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620162    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb93956-b13e-4470-8a61-979f9c6a5b07-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:19.620629 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:19.620170    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb93956-b13e-4470-8a61-979f9c6a5b07-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:20.239425 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.239387    2566 generic.go:356] "Generic (PLEG): container finished" podID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerID="678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50" exitCode=0
Sep 17 18:34:20.239645 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.239439    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:34:20.239645 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.239454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bfb93956-b13e-4470-8a61-979f9c6a5b07","Type":"ContainerDied","Data":"678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50"}
Sep 17 18:34:20.239645 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.239517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bfb93956-b13e-4470-8a61-979f9c6a5b07","Type":"ContainerDied","Data":"7f5a3095c4bfe6b1f3af162f05f7b05ae2effe8ab76d7e9a9ac113535c54bb71"}
Sep 17 18:34:20.239645 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.239548    2566 scope.go:117] "RemoveContainer" containerID="678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50"
Sep 17 18:34:20.248468 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.248447    2566 scope.go:117] "RemoveContainer" containerID="ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5"
Sep 17 18:34:20.258383 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.258359    2566 scope.go:117] "RemoveContainer" containerID="678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50"
Sep 17 18:34:20.258665 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:34:20.258648    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50\": container with ID starting with 678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50 not found: ID does not exist" containerID="678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50"
Sep 17 18:34:20.258720 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.258674    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50"} err="failed to get container status \"678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50\": rpc error: code = NotFound desc = could not find container \"678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50\": container with ID starting with 678842d715a4af15382b91bdbe6acedfc6d32b3472a362cadc1bbcf327140f50 not found: ID does not exist"
Sep 17 18:34:20.258720 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.258693    2566 scope.go:117] "RemoveContainer" containerID="ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5"
Sep 17 18:34:20.258921 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:34:20.258897    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5\": container with ID starting with ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5 not found: ID does not exist" containerID="ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5"
Sep 17 18:34:20.258972 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.258933    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5"} err="failed to get container status \"ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5\": rpc error: code = NotFound desc = could not find container \"ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5\": container with ID starting with ae96323e0eae64975a7645ed0dd7473a3bc061a2ca0f84fe3219f874fe3212c5 not found: ID does not exist"
Sep 17 18:34:20.266185 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.266159    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:34:20.272792 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.272769    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:34:20.507089 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:20.507000    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" path="/var/lib/kubelet/pods/bfb93956-b13e-4470-8a61-979f9c6a5b07/volumes"
Sep 17 18:34:28.581734 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:28.581693    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:28.582161 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:28.581797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:34:38.582313 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:38.582266    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:48.581640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:48.581601    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:48.872281 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:48.872257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:34:48.982738 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:48.982706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f256z\" (UniqueName: \"kubernetes.io/projected/a75324d8-a2eb-4581-b9c7-93b4636446c0-kube-api-access-f256z\") pod \"a75324d8-a2eb-4581-b9c7-93b4636446c0\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") "
Sep 17 18:34:48.982942 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:48.982755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75324d8-a2eb-4581-b9c7-93b4636446c0-tls-certs\") pod \"a75324d8-a2eb-4581-b9c7-93b4636446c0\" (UID: \"a75324d8-a2eb-4581-b9c7-93b4636446c0\") "
Sep 17 18:34:48.984842 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:48.984810    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75324d8-a2eb-4581-b9c7-93b4636446c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a75324d8-a2eb-4581-b9c7-93b4636446c0" (UID: "a75324d8-a2eb-4581-b9c7-93b4636446c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:34:48.984986 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:48.984845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75324d8-a2eb-4581-b9c7-93b4636446c0-kube-api-access-f256z" (OuterVolumeSpecName: "kube-api-access-f256z") pod "a75324d8-a2eb-4581-b9c7-93b4636446c0" (UID: "a75324d8-a2eb-4581-b9c7-93b4636446c0"). InnerVolumeSpecName "kube-api-access-f256z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:49.083688 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.083655    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f256z\" (UniqueName: \"kubernetes.io/projected/a75324d8-a2eb-4581-b9c7-93b4636446c0-kube-api-access-f256z\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:49.083688 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.083685    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75324d8-a2eb-4581-b9c7-93b4636446c0-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:34:49.370258 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.370228    2566 generic.go:356] "Generic (PLEG): container finished" podID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerID="0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469" exitCode=0
Sep 17 18:34:49.370431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.370273    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"
Sep 17 18:34:49.370431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.370298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" event={"ID":"a75324d8-a2eb-4581-b9c7-93b4636446c0","Type":"ContainerDied","Data":"0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469"}
Sep 17 18:34:49.370431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.370339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7" event={"ID":"a75324d8-a2eb-4581-b9c7-93b4636446c0","Type":"ContainerDied","Data":"12f7974227d5f6305eeb548d68fd3e911fe5ece5b839333b158f4730aef45ca0"}
Sep 17 18:34:49.370431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.370362    2566 scope.go:117] "RemoveContainer" containerID="0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469"
Sep 17 18:34:49.379136 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.379119    2566 scope.go:117] "RemoveContainer" containerID="0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469"
Sep 17 18:34:49.379408 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:34:49.379385    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469\": container with ID starting with 0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469 not found: ID does not exist" containerID="0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469"
Sep 17 18:34:49.379471 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.379415    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469"} err="failed to get container status \"0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469\": rpc error: code = NotFound desc = could not find container \"0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469\": container with ID starting with 0b75ba832687cd2e5f5e987a658946d15a7c251dbd74ddcf840938606ec13469 not found: ID does not exist"
Sep 17 18:34:49.392516 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.392483    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"]
Sep 17 18:34:49.397183 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:49.397160    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6xgq7"]
Sep 17 18:34:50.506774 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:34:50.506744    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" path="/var/lib/kubelet/pods/a75324d8-a2eb-4581-b9c7-93b4636446c0/volumes"
Sep 17 18:35:04.913190 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913157    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"]
Sep 17 18:35:04.913562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913478    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="storage-initializer"
Sep 17 18:35:04.913562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913489    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="storage-initializer"
Sep 17 18:35:04.913562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913505    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main"
Sep 17 18:35:04.913562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913511    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main"
Sep 17 18:35:04.913562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913520    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main"
Sep 17 18:35:04.913562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913526    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main"
Sep 17 18:35:04.913760 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913585    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75324d8-a2eb-4581-b9c7-93b4636446c0" containerName="main"
Sep 17 18:35:04.913760 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.913595    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb93956-b13e-4470-8a61-979f9c6a5b07" containerName="main"
Sep 17 18:35:04.918272 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.918250    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:04.921635 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.921612    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 18:35:04.921752 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.921684    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-v2kqs\""
Sep 17 18:35:04.921752 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.921688    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 18:35:04.921879 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.921694    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 18:35:04.927054 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:04.927032    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"]
Sep 17 18:35:05.016433 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.016400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f791-feec-4789-b4f1-e9c75d24a6c1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.016597 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.016450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.016597 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.016507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.016597 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.016543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.016597 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.016596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.016742 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.016614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9lh\" (UniqueName: \"kubernetes.io/projected/f485f791-feec-4789-b4f1-e9c75d24a6c1-kube-api-access-fq9lh\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117247 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f791-feec-4789-b4f1-e9c75d24a6c1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117431 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117607 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117607 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9lh\" (UniqueName: \"kubernetes.io/projected/f485f791-feec-4789-b4f1-e9c75d24a6c1-kube-api-access-fq9lh\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117721 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117771 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.117866 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.117846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.119668 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.119646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.119867 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.119850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f791-feec-4789-b4f1-e9c75d24a6c1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.133792 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.133767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9lh\" (UniqueName: \"kubernetes.io/projected/f485f791-feec-4789-b4f1-e9c75d24a6c1-kube-api-access-fq9lh\") pod \"scheduler-inline-config-test-kserve-8655bc757-4g2fs\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.172665 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.172581    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"]
Sep 17 18:35:05.176746 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.176725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.179214 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.179189    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-cs8kt\""
Sep 17 18:35:05.188203 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.188175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"]
Sep 17 18:35:05.217959 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.217901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6955818-e1a7-4961-a5bc-0c12a967227c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.218120 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.218018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdp2f\" (UniqueName: \"kubernetes.io/projected/b6955818-e1a7-4961-a5bc-0c12a967227c-kube-api-access-gdp2f\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.228300 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.228277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:05.318787 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.318754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdp2f\" (UniqueName: \"kubernetes.io/projected/b6955818-e1a7-4961-a5bc-0c12a967227c-kube-api-access-gdp2f\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.318981 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.318808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6955818-e1a7-4961-a5bc-0c12a967227c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.321274 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.321249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6955818-e1a7-4961-a5bc-0c12a967227c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.328638 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.328614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdp2f\" (UniqueName: \"kubernetes.io/projected/b6955818-e1a7-4961-a5bc-0c12a967227c-kube-api-access-gdp2f\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.487242 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.487167    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:05.496982 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.496954    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"]
Sep 17 18:35:05.499313 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:35:05.499273    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485f791_feec_4789_b4f1_e9c75d24a6c1.slice/crio-694fc09609559d26db955aafff8aa98ba37d32dbd5700454f337ce559dec0842 WatchSource:0}: Error finding container 694fc09609559d26db955aafff8aa98ba37d32dbd5700454f337ce559dec0842: Status 404 returned error can't find the container with id 694fc09609559d26db955aafff8aa98ba37d32dbd5700454f337ce559dec0842
Sep 17 18:35:05.772175 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:05.772150    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"]
Sep 17 18:35:05.774431 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:35:05.774406    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6955818_e1a7_4961_a5bc_0c12a967227c.slice/crio-e3a8c3222363bd7b97111cfff52a0e9192b5ef4cff9d7d3e897a124eab23f84f WatchSource:0}: Error finding container e3a8c3222363bd7b97111cfff52a0e9192b5ef4cff9d7d3e897a124eab23f84f: Status 404 returned error can't find the container with id e3a8c3222363bd7b97111cfff52a0e9192b5ef4cff9d7d3e897a124eab23f84f
Sep 17 18:35:06.452350 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:06.452311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" event={"ID":"b6955818-e1a7-4961-a5bc-0c12a967227c","Type":"ContainerStarted","Data":"30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576"}
Sep 17 18:35:06.452802 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:06.452405    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" event={"ID":"b6955818-e1a7-4961-a5bc-0c12a967227c","Type":"ContainerStarted","Data":"e3a8c3222363bd7b97111cfff52a0e9192b5ef4cff9d7d3e897a124eab23f84f"}
Sep 17 18:35:06.452802 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:06.452433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:06.454162 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:06.454134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" event={"ID":"f485f791-feec-4789-b4f1-e9c75d24a6c1","Type":"ContainerStarted","Data":"537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a"}
Sep 17 18:35:06.454278 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:06.454185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" event={"ID":"f485f791-feec-4789-b4f1-e9c75d24a6c1","Type":"ContainerStarted","Data":"694fc09609559d26db955aafff8aa98ba37d32dbd5700454f337ce559dec0842"}
Sep 17 18:35:06.473147 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:06.473102    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" podStartSLOduration=1.473090226 podStartE2EDuration="1.473090226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:06.471045106 +0000 UTC m=+3666.330636461" watchObservedRunningTime="2026-09-17 18:35:06.473090226 +0000 UTC m=+3666.332681581"
Sep 17 18:35:06.492376 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:06.492328    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" podStartSLOduration=2.492315103 podStartE2EDuration="2.492315103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:06.490404193 +0000 UTC m=+3666.349995549" watchObservedRunningTime="2026-09-17 18:35:06.492315103 +0000 UTC m=+3666.351906459"
Sep 17 18:35:07.229081 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:07.229050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:15.229176 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:15.229076    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:15.241927 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:15.241882    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:15.504395 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:15.504315    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:37.462150 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:37.462119    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:35:46.452967 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.452929    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"]
Sep 17 18:35:46.454496 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.454467    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"]
Sep 17 18:35:46.462358 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:35:46.462321    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:35:46.462358 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:35:46.462358 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:35:46.462358 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a" cmd=["/bin/sleep","15"]
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.462372    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" message="" err=<
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:35:46.462385    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" podUID="f485f791-feec-4789-b4f1-e9c75d24a6c1" containerName="main" containerID="cri-o://537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a"
Sep 17 18:35:46.462685 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.462422    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" podUID="f485f791-feec-4789-b4f1-e9c75d24a6c1" containerName="main" containerID="cri-o://537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a" gracePeriod=60
Sep 17 18:35:46.493445 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.493408    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" podUID="f485f791-feec-4789-b4f1-e9c75d24a6c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:35:46.814434 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.814404    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:46.888129 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-model-cache\") pod \"f485f791-feec-4789-b4f1-e9c75d24a6c1\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") "
Sep 17 18:35:46.888278 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f791-feec-4789-b4f1-e9c75d24a6c1-tls-certs\") pod \"f485f791-feec-4789-b4f1-e9c75d24a6c1\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") "
Sep 17 18:35:46.888278 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-dshm\") pod \"f485f791-feec-4789-b4f1-e9c75d24a6c1\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") "
Sep 17 18:35:46.888278 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888256    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq9lh\" (UniqueName: \"kubernetes.io/projected/f485f791-feec-4789-b4f1-e9c75d24a6c1-kube-api-access-fq9lh\") pod \"f485f791-feec-4789-b4f1-e9c75d24a6c1\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") "
Sep 17 18:35:46.888417 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888285    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-home\") pod \"f485f791-feec-4789-b4f1-e9c75d24a6c1\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") "
Sep 17 18:35:46.888417 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-tmp-dir\") pod \"f485f791-feec-4789-b4f1-e9c75d24a6c1\" (UID: \"f485f791-feec-4789-b4f1-e9c75d24a6c1\") "
Sep 17 18:35:46.888417 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f485f791-feec-4789-b4f1-e9c75d24a6c1" (UID: "f485f791-feec-4789-b4f1-e9c75d24a6c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:46.888577 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888559    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:35:46.888635 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888619    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f485f791-feec-4789-b4f1-e9c75d24a6c1" (UID: "f485f791-feec-4789-b4f1-e9c75d24a6c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:46.888672 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.888643    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-home" (OuterVolumeSpecName: "home") pod "f485f791-feec-4789-b4f1-e9c75d24a6c1" (UID: "f485f791-feec-4789-b4f1-e9c75d24a6c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:46.890556 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.890523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-dshm" (OuterVolumeSpecName: "dshm") pod "f485f791-feec-4789-b4f1-e9c75d24a6c1" (UID: "f485f791-feec-4789-b4f1-e9c75d24a6c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:46.890661 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.890566    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f485f791-feec-4789-b4f1-e9c75d24a6c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f485f791-feec-4789-b4f1-e9c75d24a6c1" (UID: "f485f791-feec-4789-b4f1-e9c75d24a6c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:46.890661 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.890584    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f485f791-feec-4789-b4f1-e9c75d24a6c1-kube-api-access-fq9lh" (OuterVolumeSpecName: "kube-api-access-fq9lh") pod "f485f791-feec-4789-b4f1-e9c75d24a6c1" (UID: "f485f791-feec-4789-b4f1-e9c75d24a6c1"). InnerVolumeSpecName "kube-api-access-fq9lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:46.989655 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.989562    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq9lh\" (UniqueName: \"kubernetes.io/projected/f485f791-feec-4789-b4f1-e9c75d24a6c1-kube-api-access-fq9lh\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:35:46.989655 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.989593    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:35:46.989655 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.989606    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:35:46.989655 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.989616    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f791-feec-4789-b4f1-e9c75d24a6c1-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:35:46.989655 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:46.989628    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f791-feec-4789-b4f1-e9c75d24a6c1-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:35:47.461603 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.461560    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:35:47.635381 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.635350    2566 generic.go:356] "Generic (PLEG): container finished" podID="f485f791-feec-4789-b4f1-e9c75d24a6c1" containerID="537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a" exitCode=0
Sep 17 18:35:47.635562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.635390    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"
Sep 17 18:35:47.635562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.635431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" event={"ID":"f485f791-feec-4789-b4f1-e9c75d24a6c1","Type":"ContainerDied","Data":"537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a"}
Sep 17 18:35:47.635562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.635488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs" event={"ID":"f485f791-feec-4789-b4f1-e9c75d24a6c1","Type":"ContainerDied","Data":"694fc09609559d26db955aafff8aa98ba37d32dbd5700454f337ce559dec0842"}
Sep 17 18:35:47.635562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.635516    2566 scope.go:117] "RemoveContainer" containerID="537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a"
Sep 17 18:35:47.645153 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.645134    2566 scope.go:117] "RemoveContainer" containerID="537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a"
Sep 17 18:35:47.645444 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:35:47.645422    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a\": container with ID starting with 537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a not found: ID does not exist" containerID="537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a"
Sep 17 18:35:47.645485 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.645456    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a"} err="failed to get container status \"537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a\": rpc error: code = NotFound desc = could not find container \"537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a\": container with ID starting with 537cd4c5007331723c80b883496e2b8d21ca350fa5327b8b55cfe0d4bc73661a not found: ID does not exist"
Sep 17 18:35:47.657637 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.657527    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"]
Sep 17 18:35:47.659540 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:47.659514    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4g2fs"]
Sep 17 18:35:48.506970 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:48.506939    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f485f791-feec-4789-b4f1-e9c75d24a6c1" path="/var/lib/kubelet/pods/f485f791-feec-4789-b4f1-e9c75d24a6c1/volumes"
Sep 17 18:35:57.461144 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:35:57.461107    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:01.473412 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:01.473343    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main" containerID="cri-o://30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576" gracePeriod=45
Sep 17 18:36:02.914610 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:02.914572    2566 scope.go:117] "RemoveContainer" containerID="7d61fdc2e5399ec4832c4e973b18b307d269f75abad629f764b9b7551bb99c71"
Sep 17 18:36:07.461387 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:07.461343    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:07.461759 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:07.461476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:36:17.461501 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:17.461460    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:27.461144 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:27.461109    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:31.711026 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.710995    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:36:31.768979 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.768887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6955818-e1a7-4961-a5bc-0c12a967227c-tls-certs\") pod \"b6955818-e1a7-4961-a5bc-0c12a967227c\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") "
Sep 17 18:36:31.768979 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.768957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdp2f\" (UniqueName: \"kubernetes.io/projected/b6955818-e1a7-4961-a5bc-0c12a967227c-kube-api-access-gdp2f\") pod \"b6955818-e1a7-4961-a5bc-0c12a967227c\" (UID: \"b6955818-e1a7-4961-a5bc-0c12a967227c\") "
Sep 17 18:36:31.771149 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.771117    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6955818-e1a7-4961-a5bc-0c12a967227c-kube-api-access-gdp2f" (OuterVolumeSpecName: "kube-api-access-gdp2f") pod "b6955818-e1a7-4961-a5bc-0c12a967227c" (UID: "b6955818-e1a7-4961-a5bc-0c12a967227c"). InnerVolumeSpecName "kube-api-access-gdp2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:31.771267 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.771151    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6955818-e1a7-4961-a5bc-0c12a967227c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6955818-e1a7-4961-a5bc-0c12a967227c" (UID: "b6955818-e1a7-4961-a5bc-0c12a967227c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:31.820760 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.820724    2566 generic.go:356] "Generic (PLEG): container finished" podID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerID="30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576" exitCode=0
Sep 17 18:36:31.820933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.820767    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"
Sep 17 18:36:31.820933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.820806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" event={"ID":"b6955818-e1a7-4961-a5bc-0c12a967227c","Type":"ContainerDied","Data":"30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576"}
Sep 17 18:36:31.820933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.820850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp" event={"ID":"b6955818-e1a7-4961-a5bc-0c12a967227c","Type":"ContainerDied","Data":"e3a8c3222363bd7b97111cfff52a0e9192b5ef4cff9d7d3e897a124eab23f84f"}
Sep 17 18:36:31.820933 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.820875    2566 scope.go:117] "RemoveContainer" containerID="30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576"
Sep 17 18:36:31.831345 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.831326    2566 scope.go:117] "RemoveContainer" containerID="30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576"
Sep 17 18:36:31.831670 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:36:31.831646    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576\": container with ID starting with 30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576 not found: ID does not exist" containerID="30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576"
Sep 17 18:36:31.831734 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.831685    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576"} err="failed to get container status \"30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576\": rpc error: code = NotFound desc = could not find container \"30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576\": container with ID starting with 30bcb36bd3e462d61a4ca65843ea44537dde46a38cc0af24a66436bc076af576 not found: ID does not exist"
Sep 17 18:36:31.845011 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.844985    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"]
Sep 17 18:36:31.850898 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.850873    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb72hnp"]
Sep 17 18:36:31.869573 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.869547    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6955818-e1a7-4961-a5bc-0c12a967227c-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:36:31.869573 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:31.869571    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdp2f\" (UniqueName: \"kubernetes.io/projected/b6955818-e1a7-4961-a5bc-0c12a967227c-kube-api-access-gdp2f\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:36:32.506510 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:32.506473    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" path="/var/lib/kubelet/pods/b6955818-e1a7-4961-a5bc-0c12a967227c/volumes"
Sep 17 18:36:43.423388 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.423287    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"]
Sep 17 18:36:43.423749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.423648    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f485f791-feec-4789-b4f1-e9c75d24a6c1" containerName="main"
Sep 17 18:36:43.423749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.423660    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f485f791-feec-4789-b4f1-e9c75d24a6c1" containerName="main"
Sep 17 18:36:43.423749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.423673    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main"
Sep 17 18:36:43.423749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.423679    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main"
Sep 17 18:36:43.423749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.423734    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f485f791-feec-4789-b4f1-e9c75d24a6c1" containerName="main"
Sep 17 18:36:43.423749 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.423744    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6955818-e1a7-4961-a5bc-0c12a967227c" containerName="main"
Sep 17 18:36:43.426512 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.426495    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.430153 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.430123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-zgwbf\""
Sep 17 18:36:43.430304 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.430123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 18:36:43.430304 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.430184    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 18:36:43.430491 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.430465    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 18:36:43.438478 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.438453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"]
Sep 17 18:36:43.467127 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.467095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkcq\" (UniqueName: \"kubernetes.io/projected/d1b877d7-5de3-427f-bf38-a767dca1dc4c-kube-api-access-bvkcq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.467127 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.467130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.467356 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.467153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.467356 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.467209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.467356 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.467234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.467356 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.467306    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.567794 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.567758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkcq\" (UniqueName: \"kubernetes.io/projected/d1b877d7-5de3-427f-bf38-a767dca1dc4c-kube-api-access-bvkcq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.567794 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.567796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.568074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.567812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.568074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.567944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.568074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.567987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.568074 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.568025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.568362 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.568341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.568446 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.568386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.568446 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.568409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.570205 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.570185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.570389 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.570373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.576345 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.576324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkcq\" (UniqueName: \"kubernetes.io/projected/d1b877d7-5de3-427f-bf38-a767dca1dc4c-kube-api-access-bvkcq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.646002 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.645954    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"]
Sep 17 18:36:43.654432 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.654401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:43.656990 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.656960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-njhmh\""
Sep 17 18:36:43.660443 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.660415    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"]
Sep 17 18:36:43.737710 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.737617    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:43.769827 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.769792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2cbd62-eff5-435a-851f-e9006b06d84a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:43.770029 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.769941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brl28\" (UniqueName: \"kubernetes.io/projected/8a2cbd62-eff5-435a-851f-e9006b06d84a-kube-api-access-brl28\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:43.870884 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.870850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2cbd62-eff5-435a-851f-e9006b06d84a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:43.871120 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.870967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl28\" (UniqueName: \"kubernetes.io/projected/8a2cbd62-eff5-435a-851f-e9006b06d84a-kube-api-access-brl28\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:43.873544 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.873519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2cbd62-eff5-435a-851f-e9006b06d84a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:43.879369 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.879346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl28\" (UniqueName: \"kubernetes.io/projected/8a2cbd62-eff5-435a-851f-e9006b06d84a-kube-api-access-brl28\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:43.966775 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:43.966734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:44.019113 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.019081    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"]
Sep 17 18:36:44.021511 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:36:44.021472    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b877d7_5de3_427f_bf38_a767dca1dc4c.slice/crio-395b7391afa875f4dd255c960109bffeb64e94411dbd66b5bb0cb8bb636ebb6c WatchSource:0}: Error finding container 395b7391afa875f4dd255c960109bffeb64e94411dbd66b5bb0cb8bb636ebb6c: Status 404 returned error can't find the container with id 395b7391afa875f4dd255c960109bffeb64e94411dbd66b5bb0cb8bb636ebb6c
Sep 17 18:36:44.240471 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.240328    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"]
Sep 17 18:36:44.242881 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:36:44.242858    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2cbd62_eff5_435a_851f_e9006b06d84a.slice/crio-866fab42153c3fd608703cf292c4a9daf4b05e9e48b398c2f66bbe1c0d3519bf WatchSource:0}: Error finding container 866fab42153c3fd608703cf292c4a9daf4b05e9e48b398c2f66bbe1c0d3519bf: Status 404 returned error can't find the container with id 866fab42153c3fd608703cf292c4a9daf4b05e9e48b398c2f66bbe1c0d3519bf
Sep 17 18:36:44.876708 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.876669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" event={"ID":"8a2cbd62-eff5-435a-851f-e9006b06d84a","Type":"ContainerStarted","Data":"03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504"}
Sep 17 18:36:44.877196 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.876742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" event={"ID":"8a2cbd62-eff5-435a-851f-e9006b06d84a","Type":"ContainerStarted","Data":"866fab42153c3fd608703cf292c4a9daf4b05e9e48b398c2f66bbe1c0d3519bf"}
Sep 17 18:36:44.877196 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.876777    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:36:44.878286 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.878263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" event={"ID":"d1b877d7-5de3-427f-bf38-a767dca1dc4c","Type":"ContainerStarted","Data":"46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310"}
Sep 17 18:36:44.878374 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.878305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" event={"ID":"d1b877d7-5de3-427f-bf38-a767dca1dc4c","Type":"ContainerStarted","Data":"395b7391afa875f4dd255c960109bffeb64e94411dbd66b5bb0cb8bb636ebb6c"}
Sep 17 18:36:44.894760 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.894711    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" podStartSLOduration=1.894695945 podStartE2EDuration="1.894695945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:44.89261179 +0000 UTC m=+3764.752203145" watchObservedRunningTime="2026-09-17 18:36:44.894695945 +0000 UTC m=+3764.754287301"
Sep 17 18:36:44.911850 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:44.911797    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" podStartSLOduration=1.911780556 podStartE2EDuration="1.911780556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:44.909897001 +0000 UTC m=+3764.769488358" watchObservedRunningTime="2026-09-17 18:36:44.911780556 +0000 UTC m=+3764.771371911"
Sep 17 18:36:45.737993 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:45.737966    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:53.737753 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:53.737719    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:53.750549 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:53.750517    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:36:53.928791 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:36:53.928759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:37:15.885151 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:15.885121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:37:17.519203 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.519163    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"]
Sep 17 18:37:17.521325 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.521294    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"]
Sep 17 18:37:17.531442 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:37:17.531381    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:37:17.531442 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:37:17.531442 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:37:17.531442 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310" cmd=["/bin/sleep","15"]
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.531461    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" message="" err=<
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:37:17.531479    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" podUID="d1b877d7-5de3-427f-bf38-a767dca1dc4c" containerName="main" containerID="cri-o://46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310"
Sep 17 18:37:17.531739 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.531553    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" podUID="d1b877d7-5de3-427f-bf38-a767dca1dc4c" containerName="main" containerID="cri-o://46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310" gracePeriod=60
Sep 17 18:37:17.881207 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.881182    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:37:17.978677 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.978638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tls-certs\") pod \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") "
Sep 17 18:37:17.978858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.978709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-model-cache\") pod \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") "
Sep 17 18:37:17.978858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.978749    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tmp-dir\") pod \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") "
Sep 17 18:37:17.978858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.978765    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-home\") pod \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") "
Sep 17 18:37:17.978858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.978802    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvkcq\" (UniqueName: \"kubernetes.io/projected/d1b877d7-5de3-427f-bf38-a767dca1dc4c-kube-api-access-bvkcq\") pod \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") "
Sep 17 18:37:17.978858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.978822    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-dshm\") pod \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\" (UID: \"d1b877d7-5de3-427f-bf38-a767dca1dc4c\") "
Sep 17 18:37:17.979164 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.979064    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1b877d7-5de3-427f-bf38-a767dca1dc4c" (UID: "d1b877d7-5de3-427f-bf38-a767dca1dc4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:17.979164 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.979099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1b877d7-5de3-427f-bf38-a767dca1dc4c" (UID: "d1b877d7-5de3-427f-bf38-a767dca1dc4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:17.979164 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.979119    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-home" (OuterVolumeSpecName: "home") pod "d1b877d7-5de3-427f-bf38-a767dca1dc4c" (UID: "d1b877d7-5de3-427f-bf38-a767dca1dc4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:17.981092 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.981063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-dshm" (OuterVolumeSpecName: "dshm") pod "d1b877d7-5de3-427f-bf38-a767dca1dc4c" (UID: "d1b877d7-5de3-427f-bf38-a767dca1dc4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:17.981092 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.981079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1b877d7-5de3-427f-bf38-a767dca1dc4c" (UID: "d1b877d7-5de3-427f-bf38-a767dca1dc4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:17.981269 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:17.981199    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b877d7-5de3-427f-bf38-a767dca1dc4c-kube-api-access-bvkcq" (OuterVolumeSpecName: "kube-api-access-bvkcq") pod "d1b877d7-5de3-427f-bf38-a767dca1dc4c" (UID: "d1b877d7-5de3-427f-bf38-a767dca1dc4c"). InnerVolumeSpecName "kube-api-access-bvkcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:18.024482 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.024440    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1b877d7-5de3-427f-bf38-a767dca1dc4c" containerID="46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310" exitCode=0
Sep 17 18:37:18.024640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.024488    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"
Sep 17 18:37:18.024640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.024523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" event={"ID":"d1b877d7-5de3-427f-bf38-a767dca1dc4c","Type":"ContainerDied","Data":"46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310"}
Sep 17 18:37:18.024640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.024582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr" event={"ID":"d1b877d7-5de3-427f-bf38-a767dca1dc4c","Type":"ContainerDied","Data":"395b7391afa875f4dd255c960109bffeb64e94411dbd66b5bb0cb8bb636ebb6c"}
Sep 17 18:37:18.024640 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.024604    2566 scope.go:117] "RemoveContainer" containerID="46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310"
Sep 17 18:37:18.033925 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.033883    2566 scope.go:117] "RemoveContainer" containerID="46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310"
Sep 17 18:37:18.034278 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:37:18.034250    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310\": container with ID starting with 46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310 not found: ID does not exist" containerID="46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310"
Sep 17 18:37:18.034339 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.034293    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310"} err="failed to get container status \"46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310\": rpc error: code = NotFound desc = could not find container \"46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310\": container with ID starting with 46e8fdefa7b0c211ba36ec74c07af27a82bc216ee5fea1b1fba3d7575351f310 not found: ID does not exist"
Sep 17 18:37:18.047858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.047779    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"]
Sep 17 18:37:18.050319 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.050293    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fb2rxr"]
Sep 17 18:37:18.079761 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.079734    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:37:18.079761 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.079759    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:37:18.080011 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.079769    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkcq\" (UniqueName: \"kubernetes.io/projected/d1b877d7-5de3-427f-bf38-a767dca1dc4c-kube-api-access-bvkcq\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:37:18.080011 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.079781    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:37:18.080011 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.079790    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b877d7-5de3-427f-bf38-a767dca1dc4c-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:37:18.080011 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.079799    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b877d7-5de3-427f-bf38-a767dca1dc4c-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:37:18.506739 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:18.506701    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b877d7-5de3-427f-bf38-a767dca1dc4c" path="/var/lib/kubelet/pods/d1b877d7-5de3-427f-bf38-a767dca1dc4c/volumes"
Sep 17 18:37:25.884415 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:25.884379    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:32.543013 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:32.542962    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main" containerID="cri-o://03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504" gracePeriod=45
Sep 17 18:37:35.885165 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:35.885123    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:45.885172 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:45.885129    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:45.885645 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:45.885240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:37:55.884833 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:37:55.884790    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:02.807051 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:02.806992    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:38:02.966425 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:02.966395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brl28\" (UniqueName: \"kubernetes.io/projected/8a2cbd62-eff5-435a-851f-e9006b06d84a-kube-api-access-brl28\") pod \"8a2cbd62-eff5-435a-851f-e9006b06d84a\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") "
Sep 17 18:38:02.966590 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:02.966439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2cbd62-eff5-435a-851f-e9006b06d84a-tls-certs\") pod \"8a2cbd62-eff5-435a-851f-e9006b06d84a\" (UID: \"8a2cbd62-eff5-435a-851f-e9006b06d84a\") "
Sep 17 18:38:02.968581 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:02.968555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2cbd62-eff5-435a-851f-e9006b06d84a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a2cbd62-eff5-435a-851f-e9006b06d84a" (UID: "8a2cbd62-eff5-435a-851f-e9006b06d84a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:02.968693 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:02.968601    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2cbd62-eff5-435a-851f-e9006b06d84a-kube-api-access-brl28" (OuterVolumeSpecName: "kube-api-access-brl28") pod "8a2cbd62-eff5-435a-851f-e9006b06d84a" (UID: "8a2cbd62-eff5-435a-851f-e9006b06d84a"). InnerVolumeSpecName "kube-api-access-brl28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:03.067300 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.067195    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brl28\" (UniqueName: \"kubernetes.io/projected/8a2cbd62-eff5-435a-851f-e9006b06d84a-kube-api-access-brl28\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:03.067300 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.067234    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2cbd62-eff5-435a-851f-e9006b06d84a-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:03.220450 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.220412    2566 generic.go:356] "Generic (PLEG): container finished" podID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerID="03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504" exitCode=0
Sep 17 18:38:03.220450 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.220452    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"
Sep 17 18:38:03.220666 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.220461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" event={"ID":"8a2cbd62-eff5-435a-851f-e9006b06d84a","Type":"ContainerDied","Data":"03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504"}
Sep 17 18:38:03.220666 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.220513    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt" event={"ID":"8a2cbd62-eff5-435a-851f-e9006b06d84a","Type":"ContainerDied","Data":"866fab42153c3fd608703cf292c4a9daf4b05e9e48b398c2f66bbe1c0d3519bf"}
Sep 17 18:38:03.220666 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.220537    2566 scope.go:117] "RemoveContainer" containerID="03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504"
Sep 17 18:38:03.229365 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.229346    2566 scope.go:117] "RemoveContainer" containerID="03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504"
Sep 17 18:38:03.229630 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:38:03.229611    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504\": container with ID starting with 03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504 not found: ID does not exist" containerID="03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504"
Sep 17 18:38:03.229676 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.229637    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504"} err="failed to get container status \"03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504\": rpc error: code = NotFound desc = could not find container \"03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504\": container with ID starting with 03e866ee52fae86767891895306c4220d57267b6523aafd8b73e335dd80d1504 not found: ID does not exist"
Sep 17 18:38:03.246134 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.246108    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"]
Sep 17 18:38:03.252890 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:03.252866    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheg62lt"]
Sep 17 18:38:04.506784 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:04.506749    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" path="/var/lib/kubelet/pods/8a2cbd62-eff5-435a-851f-e9006b06d84a/volumes"
Sep 17 18:38:12.605929 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.605880    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"]
Sep 17 18:38:12.606394 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.606370    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main"
Sep 17 18:38:12.606394 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.606389    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main"
Sep 17 18:38:12.606510 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.606427    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b877d7-5de3-427f-bf38-a767dca1dc4c" containerName="main"
Sep 17 18:38:12.606510 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.606437    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b877d7-5de3-427f-bf38-a767dca1dc4c" containerName="main"
Sep 17 18:38:12.606613 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.606515    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2cbd62-eff5-435a-851f-e9006b06d84a" containerName="main"
Sep 17 18:38:12.606613 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.606531    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b877d7-5de3-427f-bf38-a767dca1dc4c" containerName="main"
Sep 17 18:38:12.609552 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.609526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.612001 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.611981    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 18:38:12.612133 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.612019    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 18:38:12.613212 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.613195    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-sn2nw\""
Sep 17 18:38:12.613282 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.613261    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 18:38:12.619153 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.619129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"]
Sep 17 18:38:12.752025 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.751979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzws\" (UniqueName: \"kubernetes.io/projected/684dd6d0-0f96-426d-8380-d2d0e52d797f-kube-api-access-wkzws\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.752025 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.752021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.752322 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.752048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.752322 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.752142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.752322 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.752186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.752322 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.752226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684dd6d0-0f96-426d-8380-d2d0e52d797f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.852711 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.852669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzws\" (UniqueName: \"kubernetes.io/projected/684dd6d0-0f96-426d-8380-d2d0e52d797f-kube-api-access-wkzws\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.852711 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.852709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.853002 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.852736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.853002 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.852758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.853002 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.852820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.853002 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.852858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684dd6d0-0f96-426d-8380-d2d0e52d797f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.853281 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.853247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.853396 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.853256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.853396 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.853307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.855169 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.855142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.855579 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.855556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684dd6d0-0f96-426d-8380-d2d0e52d797f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.862325 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.862263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzws\" (UniqueName: \"kubernetes.io/projected/684dd6d0-0f96-426d-8380-d2d0e52d797f-kube-api-access-wkzws\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.920778 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.920741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:12.940224 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.940193    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"]
Sep 17 18:38:12.944338 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.944317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:12.946631 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.946609    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-r9bcd\""
Sep 17 18:38:12.950860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:12.950825    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"]
Sep 17 18:38:13.054781 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.054739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshh4\" (UniqueName: \"kubernetes.io/projected/fcb5f784-b3c3-40c9-94a4-90fe0542591a-kube-api-access-tshh4\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:13.054994 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.054850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb5f784-b3c3-40c9-94a4-90fe0542591a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:13.155951 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.155808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb5f784-b3c3-40c9-94a4-90fe0542591a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:13.155951 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.155941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshh4\" (UniqueName: \"kubernetes.io/projected/fcb5f784-b3c3-40c9-94a4-90fe0542591a-kube-api-access-tshh4\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:13.158571 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.158550    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb5f784-b3c3-40c9-94a4-90fe0542591a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:13.164852 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.164818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshh4\" (UniqueName: \"kubernetes.io/projected/fcb5f784-b3c3-40c9-94a4-90fe0542591a-kube-api-access-tshh4\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:13.202101 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.200617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"]
Sep 17 18:38:13.202835 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:38:13.202802    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684dd6d0_0f96_426d_8380_d2d0e52d797f.slice/crio-2c495f413741c9b5747dac270e40560e7986326c0cc0d594d6bd70eb5c98eb6a WatchSource:0}: Error finding container 2c495f413741c9b5747dac270e40560e7986326c0cc0d594d6bd70eb5c98eb6a: Status 404 returned error can't find the container with id 2c495f413741c9b5747dac270e40560e7986326c0cc0d594d6bd70eb5c98eb6a
Sep 17 18:38:13.255819 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.255793    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:13.262967 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.262935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" event={"ID":"684dd6d0-0f96-426d-8380-d2d0e52d797f","Type":"ContainerStarted","Data":"2c495f413741c9b5747dac270e40560e7986326c0cc0d594d6bd70eb5c98eb6a"}
Sep 17 18:38:13.534973 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:13.534946    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"]
Sep 17 18:38:13.537140 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:38:13.537110    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb5f784_b3c3_40c9_94a4_90fe0542591a.slice/crio-c6794056abe866d9003f6f6ebf73c54f5011b901dee26910740c1b26dd535f13 WatchSource:0}: Error finding container c6794056abe866d9003f6f6ebf73c54f5011b901dee26910740c1b26dd535f13: Status 404 returned error can't find the container with id c6794056abe866d9003f6f6ebf73c54f5011b901dee26910740c1b26dd535f13
Sep 17 18:38:14.268651 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:14.268614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" event={"ID":"684dd6d0-0f96-426d-8380-d2d0e52d797f","Type":"ContainerStarted","Data":"635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5"}
Sep 17 18:38:14.270149 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:14.270126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" event={"ID":"fcb5f784-b3c3-40c9-94a4-90fe0542591a","Type":"ContainerStarted","Data":"5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595"}
Sep 17 18:38:14.270253 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:14.270165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" event={"ID":"fcb5f784-b3c3-40c9-94a4-90fe0542591a","Type":"ContainerStarted","Data":"c6794056abe866d9003f6f6ebf73c54f5011b901dee26910740c1b26dd535f13"}
Sep 17 18:38:14.270253 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:14.270187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:14.286414 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:14.286365    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" podStartSLOduration=2.286351326 podStartE2EDuration="2.286351326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:14.284806949 +0000 UTC m=+3854.144398307" watchObservedRunningTime="2026-09-17 18:38:14.286351326 +0000 UTC m=+3854.145942681"
Sep 17 18:38:14.301899 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:14.301847    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" podStartSLOduration=2.3018288780000002 podStartE2EDuration="2.301828878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:14.300447813 +0000 UTC m=+3854.160039169" watchObservedRunningTime="2026-09-17 18:38:14.301828878 +0000 UTC m=+3854.161420233"
Sep 17 18:38:14.921313 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:14.921269    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:22.921566 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:22.921528    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:22.934067 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:22.934039    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:23.318586 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:23.318557    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:45.276063 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:45.276027    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:38:48.244036 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.243993    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"]
Sep 17 18:38:48.251817 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.251786    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"]
Sep 17 18:38:48.255050 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:38:48.255014    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:38:48.255050 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:38:48.255050 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:38:48.255050 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5" cmd=["/bin/sleep","15"]
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.255077    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" message="" err=<
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:38:48.255095    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" podUID="684dd6d0-0f96-426d-8380-d2d0e52d797f" containerName="main" containerID="cri-o://635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5"
Sep 17 18:38:48.255359 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.255158    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" podUID="684dd6d0-0f96-426d-8380-d2d0e52d797f" containerName="main" containerID="cri-o://635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5" gracePeriod=60
Sep 17 18:38:48.308371 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.308333    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" podUID="684dd6d0-0f96-426d-8380-d2d0e52d797f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 18:38:48.619519 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.619477    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:48.773776 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.773714    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-dshm\") pod \"684dd6d0-0f96-426d-8380-d2d0e52d797f\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") "
Sep 17 18:38:48.773776 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.773776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-tmp-dir\") pod \"684dd6d0-0f96-426d-8380-d2d0e52d797f\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") "
Sep 17 18:38:48.774096 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.773805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkzws\" (UniqueName: \"kubernetes.io/projected/684dd6d0-0f96-426d-8380-d2d0e52d797f-kube-api-access-wkzws\") pod \"684dd6d0-0f96-426d-8380-d2d0e52d797f\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") "
Sep 17 18:38:48.774096 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.773827    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-home\") pod \"684dd6d0-0f96-426d-8380-d2d0e52d797f\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") "
Sep 17 18:38:48.774096 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.773856    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-model-cache\") pod \"684dd6d0-0f96-426d-8380-d2d0e52d797f\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") "
Sep 17 18:38:48.774096 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.773904    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684dd6d0-0f96-426d-8380-d2d0e52d797f-tls-certs\") pod \"684dd6d0-0f96-426d-8380-d2d0e52d797f\" (UID: \"684dd6d0-0f96-426d-8380-d2d0e52d797f\") "
Sep 17 18:38:48.774287 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.774128    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "684dd6d0-0f96-426d-8380-d2d0e52d797f" (UID: "684dd6d0-0f96-426d-8380-d2d0e52d797f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:48.774287 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.774188    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-home" (OuterVolumeSpecName: "home") pod "684dd6d0-0f96-426d-8380-d2d0e52d797f" (UID: "684dd6d0-0f96-426d-8380-d2d0e52d797f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:48.774287 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.774224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-model-cache" (OuterVolumeSpecName: "model-cache") pod "684dd6d0-0f96-426d-8380-d2d0e52d797f" (UID: "684dd6d0-0f96-426d-8380-d2d0e52d797f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:48.776320 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.776250    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-dshm" (OuterVolumeSpecName: "dshm") pod "684dd6d0-0f96-426d-8380-d2d0e52d797f" (UID: "684dd6d0-0f96-426d-8380-d2d0e52d797f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:48.776434 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.776316    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684dd6d0-0f96-426d-8380-d2d0e52d797f-kube-api-access-wkzws" (OuterVolumeSpecName: "kube-api-access-wkzws") pod "684dd6d0-0f96-426d-8380-d2d0e52d797f" (UID: "684dd6d0-0f96-426d-8380-d2d0e52d797f"). InnerVolumeSpecName "kube-api-access-wkzws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:48.776434 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.776352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684dd6d0-0f96-426d-8380-d2d0e52d797f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "684dd6d0-0f96-426d-8380-d2d0e52d797f" (UID: "684dd6d0-0f96-426d-8380-d2d0e52d797f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:48.875078 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.875034    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:48.875078 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.875071    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684dd6d0-0f96-426d-8380-d2d0e52d797f-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:48.875078 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.875079    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:48.875333 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.875107    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:48.875333 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.875117    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzws\" (UniqueName: \"kubernetes.io/projected/684dd6d0-0f96-426d-8380-d2d0e52d797f-kube-api-access-wkzws\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:48.875333 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:48.875136    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684dd6d0-0f96-426d-8380-d2d0e52d797f-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:38:49.420958 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.420896    2566 generic.go:356] "Generic (PLEG): container finished" podID="684dd6d0-0f96-426d-8380-d2d0e52d797f" containerID="635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5" exitCode=0
Sep 17 18:38:49.420958 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.420943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" event={"ID":"684dd6d0-0f96-426d-8380-d2d0e52d797f","Type":"ContainerDied","Data":"635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5"}
Sep 17 18:38:49.421465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.420973    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"
Sep 17 18:38:49.421465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.421034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68" event={"ID":"684dd6d0-0f96-426d-8380-d2d0e52d797f","Type":"ContainerDied","Data":"2c495f413741c9b5747dac270e40560e7986326c0cc0d594d6bd70eb5c98eb6a"}
Sep 17 18:38:49.421465 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.421059    2566 scope.go:117] "RemoveContainer" containerID="635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5"
Sep 17 18:38:49.430624 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.430604    2566 scope.go:117] "RemoveContainer" containerID="635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5"
Sep 17 18:38:49.430995 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:38:49.430906    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5\": container with ID starting with 635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5 not found: ID does not exist" containerID="635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5"
Sep 17 18:38:49.431053 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.431008    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5"} err="failed to get container status \"635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5\": rpc error: code = NotFound desc = could not find container \"635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5\": container with ID starting with 635c60c8aa1d570c415327cb960c14f17b909e63922e759056ca2f3934cd8ff5 not found: ID does not exist"
Sep 17 18:38:49.444623 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.444590    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"]
Sep 17 18:38:49.447016 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:49.446990    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gnk68"]
Sep 17 18:38:50.506404 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:50.506372    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="684dd6d0-0f96-426d-8380-d2d0e52d797f" path="/var/lib/kubelet/pods/684dd6d0-0f96-426d-8380-d2d0e52d797f/volumes"
Sep 17 18:38:55.276562 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:38:55.276513    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:02.734190 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:02.734156    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:39:02.749965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:02.749939    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:39:03.271705 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:03.271666    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main" containerID="cri-o://5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595" gracePeriod=45
Sep 17 18:39:05.276609 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:05.276571    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:15.275858 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:15.275819    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:15.276353 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:15.275971    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:39:25.276132 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:25.276084    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:33.524863 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.524834    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:39:33.554495 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.554462    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb5f784-b3c3-40c9-94a4-90fe0542591a-tls-certs\") pod \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") "
Sep 17 18:39:33.554664 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.554587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tshh4\" (UniqueName: \"kubernetes.io/projected/fcb5f784-b3c3-40c9-94a4-90fe0542591a-kube-api-access-tshh4\") pod \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\" (UID: \"fcb5f784-b3c3-40c9-94a4-90fe0542591a\") "
Sep 17 18:39:33.556675 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.556637    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb5f784-b3c3-40c9-94a4-90fe0542591a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcb5f784-b3c3-40c9-94a4-90fe0542591a" (UID: "fcb5f784-b3c3-40c9-94a4-90fe0542591a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:33.556675 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.556653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb5f784-b3c3-40c9-94a4-90fe0542591a-kube-api-access-tshh4" (OuterVolumeSpecName: "kube-api-access-tshh4") pod "fcb5f784-b3c3-40c9-94a4-90fe0542591a" (UID: "fcb5f784-b3c3-40c9-94a4-90fe0542591a"). InnerVolumeSpecName "kube-api-access-tshh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:33.609735 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.609693    2566 generic.go:356] "Generic (PLEG): container finished" podID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerID="5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595" exitCode=0
Sep 17 18:39:33.609965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.609747    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"
Sep 17 18:39:33.609965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.609787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" event={"ID":"fcb5f784-b3c3-40c9-94a4-90fe0542591a","Type":"ContainerDied","Data":"5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595"}
Sep 17 18:39:33.609965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.609881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn" event={"ID":"fcb5f784-b3c3-40c9-94a4-90fe0542591a","Type":"ContainerDied","Data":"c6794056abe866d9003f6f6ebf73c54f5011b901dee26910740c1b26dd535f13"}
Sep 17 18:39:33.609965 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.609924    2566 scope.go:117] "RemoveContainer" containerID="5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595"
Sep 17 18:39:33.618654 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.618637    2566 scope.go:117] "RemoveContainer" containerID="5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595"
Sep 17 18:39:33.618962 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:39:33.618941    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595\": container with ID starting with 5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595 not found: ID does not exist" containerID="5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595"
Sep 17 18:39:33.619057 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.618976    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595"} err="failed to get container status \"5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595\": rpc error: code = NotFound desc = could not find container \"5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595\": container with ID starting with 5e41a5b46d81e1ef28c8f8a2f2dc5eb6b342c68a5061de7d7a1908a44bcb5595 not found: ID does not exist"
Sep 17 18:39:33.632404 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.632375    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"]
Sep 17 18:39:33.639658 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.639635    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9ln8xn"]
Sep 17 18:39:33.656288 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.656257    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tshh4\" (UniqueName: \"kubernetes.io/projected/fcb5f784-b3c3-40c9-94a4-90fe0542591a-kube-api-access-tshh4\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:39:33.656288 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:33.656287    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb5f784-b3c3-40c9-94a4-90fe0542591a-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:39:34.506788 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:34.506751    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" path="/var/lib/kubelet/pods/fcb5f784-b3c3-40c9-94a4-90fe0542591a/volumes"
Sep 17 18:39:50.711904 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.711835    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"]
Sep 17 18:39:50.712373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.712156    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main"
Sep 17 18:39:50.712373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.712169    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main"
Sep 17 18:39:50.712373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.712185    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684dd6d0-0f96-426d-8380-d2d0e52d797f" containerName="main"
Sep 17 18:39:50.712373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.712190    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="684dd6d0-0f96-426d-8380-d2d0e52d797f" containerName="main"
Sep 17 18:39:50.712373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.712245    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb5f784-b3c3-40c9-94a4-90fe0542591a" containerName="main"
Sep 17 18:39:50.712373 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.712253    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="684dd6d0-0f96-426d-8380-d2d0e52d797f" containerName="main"
Sep 17 18:39:50.715119 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.715102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.718982 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.718956    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jlcr5\""
Sep 17 18:39:50.718982 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.718971    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 18:39:50.719168 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.718974    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 18:39:50.719168 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.719024    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 18:39:50.724712 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.724689    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"]
Sep 17 18:39:50.806850 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.806819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.807035 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.806865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.807035 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.806884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.807035 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.806994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.807035 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.807024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.807199 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.807043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8zx\" (UniqueName: \"kubernetes.io/projected/f00484a9-bfcb-4f70-89bf-6e9247a946a4-kube-api-access-xj8zx\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908278 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908454 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908454 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908454 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8zx\" (UniqueName: \"kubernetes.io/projected/f00484a9-bfcb-4f70-89bf-6e9247a946a4-kube-api-access-xj8zx\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908678 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908678 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908820 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908881 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.908981 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.908955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.910730 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.910698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.911160 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.911142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.917527 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.917503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8zx\" (UniqueName: \"kubernetes.io/projected/f00484a9-bfcb-4f70-89bf-6e9247a946a4-kube-api-access-xj8zx\") pod \"rolling-upgrade-test-kserve-74b79d7f77-w9v9j\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:50.982419 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.982343    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"]
Sep 17 18:39:50.986090 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.986069    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:50.988394 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.988373    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-xml9v\""
Sep 17 18:39:50.995423 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:50.995397    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"]
Sep 17 18:39:51.026068 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.026044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:39:51.110564 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.110533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpd6v\" (UniqueName: \"kubernetes.io/projected/ae6f2050-974f-4a54-937c-002d4bc80f47-kube-api-access-bpd6v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.110722 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.110597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f2050-974f-4a54-937c-002d4bc80f47-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.211995 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.211959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f2050-974f-4a54-937c-002d4bc80f47-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.212159 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.212017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpd6v\" (UniqueName: \"kubernetes.io/projected/ae6f2050-974f-4a54-937c-002d4bc80f47-kube-api-access-bpd6v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.214612 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.214588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f2050-974f-4a54-937c-002d4bc80f47-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.221596 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.221572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpd6v\" (UniqueName: \"kubernetes.io/projected/ae6f2050-974f-4a54-937c-002d4bc80f47-kube-api-access-bpd6v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.296402 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.296346    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"]
Sep 17 18:39:51.297486 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.297466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.298759 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:39:51.298732    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00484a9_bfcb_4f70_89bf_6e9247a946a4.slice/crio-4803f68a3c48d198f10ad2f4fb21a9a00b76cc07c6c5df34d70ab422f3b81890 WatchSource:0}: Error finding container 4803f68a3c48d198f10ad2f4fb21a9a00b76cc07c6c5df34d70ab422f3b81890: Status 404 returned error can't find the container with id 4803f68a3c48d198f10ad2f4fb21a9a00b76cc07c6c5df34d70ab422f3b81890
Sep 17 18:39:51.568945 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.568878    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"]
Sep 17 18:39:51.571090 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:39:51.571062    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6f2050_974f_4a54_937c_002d4bc80f47.slice/crio-8f62ef22b8ed4080445b80bb2498acda5f13d67893200069c13ec7a4f93e678e WatchSource:0}: Error finding container 8f62ef22b8ed4080445b80bb2498acda5f13d67893200069c13ec7a4f93e678e: Status 404 returned error can't find the container with id 8f62ef22b8ed4080445b80bb2498acda5f13d67893200069c13ec7a4f93e678e
Sep 17 18:39:51.686897 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.686859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" event={"ID":"ae6f2050-974f-4a54-937c-002d4bc80f47","Type":"ContainerStarted","Data":"32ebd7faa87b8aa6f24191f429947deb326c727368bdcdca919219ce492db1f0"}
Sep 17 18:39:51.687092 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.686938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" event={"ID":"ae6f2050-974f-4a54-937c-002d4bc80f47","Type":"ContainerStarted","Data":"8f62ef22b8ed4080445b80bb2498acda5f13d67893200069c13ec7a4f93e678e"}
Sep 17 18:39:51.687092 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.686981    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:39:51.688453 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.688431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" event={"ID":"f00484a9-bfcb-4f70-89bf-6e9247a946a4","Type":"ContainerStarted","Data":"631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a"}
Sep 17 18:39:51.688574 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.688472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" event={"ID":"f00484a9-bfcb-4f70-89bf-6e9247a946a4","Type":"ContainerStarted","Data":"4803f68a3c48d198f10ad2f4fb21a9a00b76cc07c6c5df34d70ab422f3b81890"}
Sep 17 18:39:51.713771 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.713718    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" podStartSLOduration=1.71370147 podStartE2EDuration="1.71370147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:51.71180389 +0000 UTC m=+3951.571395246" watchObservedRunningTime="2026-09-17 18:39:51.71370147 +0000 UTC m=+3951.573292826"
Sep 17 18:39:51.741956 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:51.741878    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" podStartSLOduration=1.7418598809999999 podStartE2EDuration="1.741859881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:51.739971254 +0000 UTC m=+3951.599562603" watchObservedRunningTime="2026-09-17 18:39:51.741859881 +0000 UTC m=+3951.601451237"
Sep 17 18:39:52.026753 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:39:52.026660    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:40:01.026559 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:01.026522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:40:01.039206 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:01.039171    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:40:01.743339 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:01.743310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:40:22.694458 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:22.694426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:40:35.387188 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.387153    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"]
Sep 17 18:40:35.396657 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:40:35.396612    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:40:35.396657 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:35.396657 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:35.396657 ip-10-0-135-84 kubenswrapper[2566]:  > containerID="631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a" cmd=["/bin/sleep","15"]
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.396674    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" message="" err=<
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]:  >
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:40:35.396686    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" podUID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" containerName="main" containerID="cri-o://631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a"
Sep 17 18:40:35.396958 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.396731    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" podUID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" containerName="main" containerID="cri-o://631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a" gracePeriod=60
Sep 17 18:40:35.748568 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.748540    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:40:35.808960 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.808899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-home\") pod \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") "
Sep 17 18:40:35.809138 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.808998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tmp-dir\") pod \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") "
Sep 17 18:40:35.809138 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809022    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-model-cache\") pod \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") "
Sep 17 18:40:35.809138 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809047    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tls-certs\") pod \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") "
Sep 17 18:40:35.809138 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-dshm\") pod \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") "
Sep 17 18:40:35.809138 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj8zx\" (UniqueName: \"kubernetes.io/projected/f00484a9-bfcb-4f70-89bf-6e9247a946a4-kube-api-access-xj8zx\") pod \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\" (UID: \"f00484a9-bfcb-4f70-89bf-6e9247a946a4\") "
Sep 17 18:40:35.809345 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809201    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-home" (OuterVolumeSpecName: "home") pod "f00484a9-bfcb-4f70-89bf-6e9247a946a4" (UID: "f00484a9-bfcb-4f70-89bf-6e9247a946a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:35.809345 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809305    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f00484a9-bfcb-4f70-89bf-6e9247a946a4" (UID: "f00484a9-bfcb-4f70-89bf-6e9247a946a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:35.809450 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809315    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "f00484a9-bfcb-4f70-89bf-6e9247a946a4" (UID: "f00484a9-bfcb-4f70-89bf-6e9247a946a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:35.809450 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809412    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:40:35.809450 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809433    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:40:35.809450 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.809447    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:40:35.811525 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.811489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-dshm" (OuterVolumeSpecName: "dshm") pod "f00484a9-bfcb-4f70-89bf-6e9247a946a4" (UID: "f00484a9-bfcb-4f70-89bf-6e9247a946a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:35.811674 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.811529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00484a9-bfcb-4f70-89bf-6e9247a946a4-kube-api-access-xj8zx" (OuterVolumeSpecName: "kube-api-access-xj8zx") pod "f00484a9-bfcb-4f70-89bf-6e9247a946a4" (UID: "f00484a9-bfcb-4f70-89bf-6e9247a946a4"). InnerVolumeSpecName "kube-api-access-xj8zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:35.811674 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.811605    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f00484a9-bfcb-4f70-89bf-6e9247a946a4" (UID: "f00484a9-bfcb-4f70-89bf-6e9247a946a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:35.880712 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.880673    2566 generic.go:356] "Generic (PLEG): container finished" podID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" containerID="631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a" exitCode=0
Sep 17 18:40:35.880890 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.880726    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"
Sep 17 18:40:35.880890 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.880764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" event={"ID":"f00484a9-bfcb-4f70-89bf-6e9247a946a4","Type":"ContainerDied","Data":"631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a"}
Sep 17 18:40:35.880890 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.880836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" event={"ID":"f00484a9-bfcb-4f70-89bf-6e9247a946a4","Type":"ContainerDied","Data":"4803f68a3c48d198f10ad2f4fb21a9a00b76cc07c6c5df34d70ab422f3b81890"}
Sep 17 18:40:35.880890 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.880867    2566 scope.go:117] "RemoveContainer" containerID="631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a"
Sep 17 18:40:35.891043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.890993    2566 scope.go:117] "RemoveContainer" containerID="631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a"
Sep 17 18:40:35.891451 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:40:35.891423    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a\": container with ID starting with 631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a not found: ID does not exist" containerID="631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a"
Sep 17 18:40:35.891534 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.891473    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a"} err="failed to get container status \"631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a\": rpc error: code = NotFound desc = could not find container \"631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a\": container with ID starting with 631ae9e3ab046e3996e8286de20fc1fe48d0f1b5e9a0b66b336db722aa81d25a not found: ID does not exist"
Sep 17 18:40:35.909950 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.909900    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00484a9-bfcb-4f70-89bf-6e9247a946a4-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:40:35.909950 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.909951    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj8zx\" (UniqueName: \"kubernetes.io/projected/f00484a9-bfcb-4f70-89bf-6e9247a946a4-kube-api-access-xj8zx\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:40:35.910213 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.909965    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00484a9-bfcb-4f70-89bf-6e9247a946a4-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:40:35.922566 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.922480    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"]
Sep 17 18:40:35.939941 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.939898    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j"]
Sep 17 18:40:35.964293 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:35.964258    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"]
Sep 17 18:40:36.506771 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.506728    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" path="/var/lib/kubelet/pods/f00484a9-bfcb-4f70-89bf-6e9247a946a4/volumes"
Sep 17 18:40:36.732454 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.732386    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-w9v9j" podUID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:40:36.911844 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.911810    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"]
Sep 17 18:40:36.912182 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.912169    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" containerName="main"
Sep 17 18:40:36.912228 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.912183    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" containerName="main"
Sep 17 18:40:36.912263 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.912245    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00484a9-bfcb-4f70-89bf-6e9247a946a4" containerName="main"
Sep 17 18:40:36.916798 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.916781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:36.919105 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.919077    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 18:40:36.919248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.919117    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-fbrr8\""
Sep 17 18:40:36.919308 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.919258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:40:36.919308 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.919268    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:40:36.919308 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.919284    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:40:36.919435 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.919380    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:40:36.932657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:36.932629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"]
Sep 17 18:40:37.019623 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-cabundle-cert\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.019623 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227ws\" (UniqueName: \"kubernetes.io/projected/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kube-api-access-227ws\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.019893 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tmp-dir\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.019893 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-dshm\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.019893 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-home\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.019893 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kserve-provision-location\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.019893 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-model-cache\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.020121 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.019900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tls-certs\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121081 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-cabundle-cert\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227ws\" (UniqueName: \"kubernetes.io/projected/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kube-api-access-227ws\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tmp-dir\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-dshm\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-home\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121248 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kserve-provision-location\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121507 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-model-cache\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121507 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tls-certs\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121686 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-model-cache\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121818 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kserve-provision-location\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121818 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-home\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tmp-dir\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.121966 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.121937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-cabundle-cert\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.123688 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.123666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-dshm\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.123872 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.123856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tls-certs\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.131755 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.131728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-227ws\" (UniqueName: \"kubernetes.io/projected/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kube-api-access-227ws\") pod \"prestop-hook-test-kserve-7dc56b5d48-8r4l9\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.227195 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.227102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:37.524189 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.509953    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"]
Sep 17 18:40:37.524189 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:40:37.511797    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0f3fb7_8bef_4bc7_bc12_98c08849e02d.slice/crio-83e6b0c95dd13d45e6c85af1cc5d4c0afbb77b8c592f42afa5fc9a7a4d171219 WatchSource:0}: Error finding container 83e6b0c95dd13d45e6c85af1cc5d4c0afbb77b8c592f42afa5fc9a7a4d171219: Status 404 returned error can't find the container with id 83e6b0c95dd13d45e6c85af1cc5d4c0afbb77b8c592f42afa5fc9a7a4d171219
Sep 17 18:40:37.892309 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.892268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" event={"ID":"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d","Type":"ContainerStarted","Data":"aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e"}
Sep 17 18:40:37.892499 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:37.892327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" event={"ID":"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d","Type":"ContainerStarted","Data":"83e6b0c95dd13d45e6c85af1cc5d4c0afbb77b8c592f42afa5fc9a7a4d171219"}
Sep 17 18:40:40.906662 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:40.906624    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerID="aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e" exitCode=0
Sep 17 18:40:40.907221 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:40.906672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" event={"ID":"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d","Type":"ContainerDied","Data":"aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e"}
Sep 17 18:40:41.913850 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:41.913813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" event={"ID":"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d","Type":"ContainerStarted","Data":"f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76"}
Sep 17 18:40:41.936675 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:41.936620    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" podStartSLOduration=3.594703099 podStartE2EDuration="5.936604582s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.341901483s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:41.935508969 +0000 UTC m=+4001.795100325" watchObservedRunningTime="2026-09-17 18:40:41.936604582 +0000 UTC m=+4001.796195930"
Sep 17 18:40:42.227848 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:42.227761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:42.694799 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:42.694757    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:40:47.228106 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:47.228064    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:40:47.230098 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:47.230062    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:40:50.983603 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:50.983558    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main" containerID="cri-o://32ebd7faa87b8aa6f24191f429947deb326c727368bdcdca919219ce492db1f0" gracePeriod=45
Sep 17 18:40:52.695250 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:52.695205    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:40:57.228563 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:40:57.228513    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:41:02.695213 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:02.695169    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:02.695619 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:02.695281    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:41:07.228115 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:07.228064    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:41:12.695183 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:12.695145    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:13.634979 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:13.634891    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"]
Sep 17 18:41:21.099550 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.099516    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerID="32ebd7faa87b8aa6f24191f429947deb326c727368bdcdca919219ce492db1f0" exitCode=0
Sep 17 18:41:21.099885 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.099599    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" event={"ID":"ae6f2050-974f-4a54-937c-002d4bc80f47","Type":"ContainerDied","Data":"32ebd7faa87b8aa6f24191f429947deb326c727368bdcdca919219ce492db1f0"}
Sep 17 18:41:21.241664 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.241631    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:41:21.317542 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.317505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpd6v\" (UniqueName: \"kubernetes.io/projected/ae6f2050-974f-4a54-937c-002d4bc80f47-kube-api-access-bpd6v\") pod \"ae6f2050-974f-4a54-937c-002d4bc80f47\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") "
Sep 17 18:41:21.317721 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.317583    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f2050-974f-4a54-937c-002d4bc80f47-tls-certs\") pod \"ae6f2050-974f-4a54-937c-002d4bc80f47\" (UID: \"ae6f2050-974f-4a54-937c-002d4bc80f47\") "
Sep 17 18:41:21.319743 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.319713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6f2050-974f-4a54-937c-002d4bc80f47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae6f2050-974f-4a54-937c-002d4bc80f47" (UID: "ae6f2050-974f-4a54-937c-002d4bc80f47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:21.319869 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.319741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6f2050-974f-4a54-937c-002d4bc80f47-kube-api-access-bpd6v" (OuterVolumeSpecName: "kube-api-access-bpd6v") pod "ae6f2050-974f-4a54-937c-002d4bc80f47" (UID: "ae6f2050-974f-4a54-937c-002d4bc80f47"). InnerVolumeSpecName "kube-api-access-bpd6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:21.418703 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.418655    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpd6v\" (UniqueName: \"kubernetes.io/projected/ae6f2050-974f-4a54-937c-002d4bc80f47-kube-api-access-bpd6v\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:21.418703 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:21.418694    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f2050-974f-4a54-937c-002d4bc80f47-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.105560 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:22.105524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs" event={"ID":"ae6f2050-974f-4a54-937c-002d4bc80f47","Type":"ContainerDied","Data":"8f62ef22b8ed4080445b80bb2498acda5f13d67893200069c13ec7a4f93e678e"}
Sep 17 18:41:22.105560 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:22.105553    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"
Sep 17 18:41:22.106071 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:22.105592    2566 scope.go:117] "RemoveContainer" containerID="32ebd7faa87b8aa6f24191f429947deb326c727368bdcdca919219ce492db1f0"
Sep 17 18:41:22.129386 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:22.129350    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"]
Sep 17 18:41:22.134474 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:22.134444    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kd4bs"]
Sep 17 18:41:22.509188 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:22.509097    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" path="/var/lib/kubelet/pods/ae6f2050-974f-4a54-937c-002d4bc80f47/volumes"
Sep 17 18:41:28.658213 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:28.658142    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="main" containerID="cri-o://f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76" gracePeriod=45
Sep 17 18:41:30.710657 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.710631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7dc56b5d48-8r4l9_8f0f3fb7-8bef-4bc7-bc12-98c08849e02d/main/0.log"
Sep 17 18:41:30.713370 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.713350    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:41:30.805842 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.805812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kserve-provision-location\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806046 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.805863    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tls-certs\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806046 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.805897    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-227ws\" (UniqueName: \"kubernetes.io/projected/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kube-api-access-227ws\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806046 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.805950    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-home\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806046 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.805971    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-cabundle-cert\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806046 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.805999    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-dshm\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806046 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.806035    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-model-cache\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806376 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.806093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tmp-dir\") pod \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\" (UID: \"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d\") "
Sep 17 18:41:30.806376 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.806292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-home" (OuterVolumeSpecName: "home") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:30.806502 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.806419    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:30.806502 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.806459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:30.806603 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.806562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:30.808382 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.808355    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:30.808484 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.808449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kube-api-access-227ws" (OuterVolumeSpecName: "kube-api-access-227ws") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "kube-api-access-227ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:30.808524 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.808506    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-dshm" (OuterVolumeSpecName: "dshm") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:30.830234 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.830171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" (UID: "8f0f3fb7-8bef-4bc7-bc12-98c08849e02d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:30.907040 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.906996    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-home\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.907040 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.907030    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-cabundle-cert\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.907040 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.907042    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-dshm\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.907040 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.907050    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-model-cache\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.907324 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.907058    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tmp-dir\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.907324 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.907066    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kserve-provision-location\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.907324 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.907075    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-tls-certs\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.907324 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:30.907083    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-227ws\" (UniqueName: \"kubernetes.io/projected/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d-kube-api-access-227ws\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:41:31.147860 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.147780    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7dc56b5d48-8r4l9_8f0f3fb7-8bef-4bc7-bc12-98c08849e02d/main/0.log"
Sep 17 18:41:31.148263 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.148236    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerID="f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76" exitCode=130
Sep 17 18:41:31.148339 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.148299    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"
Sep 17 18:41:31.148339 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.148322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" event={"ID":"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d","Type":"ContainerDied","Data":"f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76"}
Sep 17 18:41:31.148411 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.148380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9" event={"ID":"8f0f3fb7-8bef-4bc7-bc12-98c08849e02d","Type":"ContainerDied","Data":"83e6b0c95dd13d45e6c85af1cc5d4c0afbb77b8c592f42afa5fc9a7a4d171219"}
Sep 17 18:41:31.148411 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.148403    2566 scope.go:117] "RemoveContainer" containerID="f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76"
Sep 17 18:41:31.157824 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.157799    2566 scope.go:117] "RemoveContainer" containerID="aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e"
Sep 17 18:41:31.166123 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.166101    2566 scope.go:117] "RemoveContainer" containerID="f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76"
Sep 17 18:41:31.166451 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:41:31.166430    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76\": container with ID starting with f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76 not found: ID does not exist" containerID="f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76"
Sep 17 18:41:31.166605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.166468    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76"} err="failed to get container status \"f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76\": rpc error: code = NotFound desc = could not find container \"f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76\": container with ID starting with f7f335b4604f202ba1c888f1b44771280962d43a02636e96f579923faa17ba76 not found: ID does not exist"
Sep 17 18:41:31.166605 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.166496    2566 scope.go:117] "RemoveContainer" containerID="aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e"
Sep 17 18:41:31.166775 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:41:31.166758    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e\": container with ID starting with aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e not found: ID does not exist" containerID="aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e"
Sep 17 18:41:31.166831 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.166783    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e"} err="failed to get container status \"aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e\": rpc error: code = NotFound desc = could not find container \"aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e\": container with ID starting with aa2c68d82ec399a862c4ce0f7dbb0c37c66f2f59b4cb5338b1eaf710f14c970e not found: ID does not exist"
Sep 17 18:41:31.173079 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.173044    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"]
Sep 17 18:41:31.177053 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:31.177027    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-8r4l9"]
Sep 17 18:41:32.507871 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:41:32.507837    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" path="/var/lib/kubelet/pods/8f0f3fb7-8bef-4bc7-bc12-98c08849e02d/volumes"
Sep 17 18:42:02.243264 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243224    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kxpz9/must-gather-m98rv"]
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243585    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="main"
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243598    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="main"
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243613    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="storage-initializer"
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243619    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="storage-initializer"
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243631    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main"
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243637    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main"
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243688    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0f3fb7-8bef-4bc7-bc12-98c08849e02d" containerName="main"
Sep 17 18:42:02.243780 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.243699    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6f2050-974f-4a54-937c-002d4bc80f47" containerName="main"
Sep 17 18:42:02.246737 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.246720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.249853 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.249828    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxpz9\"/\"kube-root-ca.crt\""
Sep 17 18:42:02.250036 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.249975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxpz9\"/\"openshift-service-ca.crt\""
Sep 17 18:42:02.250036 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.249986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kxpz9\"/\"default-dockercfg-vvqbp\""
Sep 17 18:42:02.256401 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.256378    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxpz9/must-gather-m98rv"]
Sep 17 18:42:02.272455 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.272409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-must-gather-output\") pod \"must-gather-m98rv\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") " pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.272625 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.272460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9k2\" (UniqueName: \"kubernetes.io/projected/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-kube-api-access-7t9k2\") pod \"must-gather-m98rv\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") " pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.372981 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.372938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-must-gather-output\") pod \"must-gather-m98rv\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") " pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.372981 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.372980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9k2\" (UniqueName: \"kubernetes.io/projected/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-kube-api-access-7t9k2\") pod \"must-gather-m98rv\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") " pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.373285 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.373265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-must-gather-output\") pod \"must-gather-m98rv\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") " pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.383854 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.383821    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxpz9\"/\"kube-root-ca.crt\""
Sep 17 18:42:02.394174 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.394145    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxpz9\"/\"openshift-service-ca.crt\""
Sep 17 18:42:02.404570 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.404540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9k2\" (UniqueName: \"kubernetes.io/projected/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-kube-api-access-7t9k2\") pod \"must-gather-m98rv\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") " pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.558717 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.558689    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kxpz9\"/\"default-dockercfg-vvqbp\""
Sep 17 18:42:02.566669 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.566641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:02.839058 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.838979    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxpz9/must-gather-m98rv"]
Sep 17 18:42:02.841650 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:42:02.841619    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf92d07d7_15f2_4a71_b4ae_188ac7a7af73.slice/crio-da76331d71c249c285d15350ef7198b29c3e842fcdc82a778e1a529f482e4fac WatchSource:0}: Error finding container da76331d71c249c285d15350ef7198b29c3e842fcdc82a778e1a529f482e4fac: Status 404 returned error can't find the container with id da76331d71c249c285d15350ef7198b29c3e842fcdc82a778e1a529f482e4fac
Sep 17 18:42:02.843722 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:02.843699    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:42:03.285247 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:03.285152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxpz9/must-gather-m98rv" event={"ID":"f92d07d7-15f2-4a71-b4ae-188ac7a7af73","Type":"ContainerStarted","Data":"da76331d71c249c285d15350ef7198b29c3e842fcdc82a778e1a529f482e4fac"}
Sep 17 18:42:10.318280 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:10.318240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxpz9/must-gather-m98rv" event={"ID":"f92d07d7-15f2-4a71-b4ae-188ac7a7af73","Type":"ContainerStarted","Data":"0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e"}
Sep 17 18:42:10.318676 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:10.318292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxpz9/must-gather-m98rv" event={"ID":"f92d07d7-15f2-4a71-b4ae-188ac7a7af73","Type":"ContainerStarted","Data":"5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665"}
Sep 17 18:42:10.335140 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:10.335075    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kxpz9/must-gather-m98rv" podStartSLOduration=1.87910953 podStartE2EDuration="8.335056695s" totalImagesPullingTime="6.455947165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:10.333707148 +0000 UTC m=+4090.193298504" watchObservedRunningTime="2026-09-17 18:42:10.335056695 +0000 UTC m=+4090.194648051"
Sep 17 18:42:26.703420 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:26.703390    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f955969b6-hrfzg_3417e40a-cdbd-4fd3-a68f-acca57237487/router/0.log"
Sep 17 18:42:27.484181 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:27.484151    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f955969b6-hrfzg_3417e40a-cdbd-4fd3-a68f-acca57237487/router/0.log"
Sep 17 18:42:33.430695 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:33.430659    2566 generic.go:356] "Generic (PLEG): container finished" podID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerID="5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665" exitCode=0
Sep 17 18:42:33.431139 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:33.430733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxpz9/must-gather-m98rv" event={"ID":"f92d07d7-15f2-4a71-b4ae-188ac7a7af73","Type":"ContainerDied","Data":"5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665"}
Sep 17 18:42:33.431139 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:33.431090    2566 scope.go:117] "RemoveContainer" containerID="5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665"
Sep 17 18:42:33.656312 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:33.656275    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kxpz9_must-gather-m98rv_f92d07d7-15f2-4a71-b4ae-188ac7a7af73/gather/0.log"
Sep 17 18:42:37.152400 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:37.152359    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8gpqj_4eccb060-886d-4453-aa0f-2160986e0827/global-pull-secret-syncer/0.log"
Sep 17 18:42:37.304565 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:37.304527    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tpz2m_541d06dd-ac25-4278-9164-bd5eb107c596/konnectivity-agent/0.log"
Sep 17 18:42:37.414191 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:37.414113    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-84.ec2.internal_af7109a8d4ffd33a7b7c4fc7c44515af/haproxy/0.log"
Sep 17 18:42:39.050805 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.050768    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-kxpz9/must-gather-m98rv"]
Sep 17 18:42:39.051261 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.050970    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-kxpz9/must-gather-m98rv" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerName="copy" containerID="cri-o://0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e" gracePeriod=2
Sep 17 18:42:39.055574 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.055102    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-kxpz9/must-gather-m98rv"]
Sep 17 18:42:39.292208 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.292177    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kxpz9_must-gather-m98rv_f92d07d7-15f2-4a71-b4ae-188ac7a7af73/copy/0.log"
Sep 17 18:42:39.293828 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.293806    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:39.297430 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:42:39.297399    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-m98rv\" is forbidden: User \"system:node:ip-10-0-135-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kxpz9\": no relationship found between node 'ip-10-0-135-84.ec2.internal' and this object" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:39.410280 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.410173    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-must-gather-output\") pod \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") "
Sep 17 18:42:39.410280 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.410259    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t9k2\" (UniqueName: \"kubernetes.io/projected/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-kube-api-access-7t9k2\") pod \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\" (UID: \"f92d07d7-15f2-4a71-b4ae-188ac7a7af73\") "
Sep 17 18:42:39.412680 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.412642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-kube-api-access-7t9k2" (OuterVolumeSpecName: "kube-api-access-7t9k2") pod "f92d07d7-15f2-4a71-b4ae-188ac7a7af73" (UID: "f92d07d7-15f2-4a71-b4ae-188ac7a7af73"). InnerVolumeSpecName "kube-api-access-7t9k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:39.416564 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.416531    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f92d07d7-15f2-4a71-b4ae-188ac7a7af73" (UID: "f92d07d7-15f2-4a71-b4ae-188ac7a7af73"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:39.458103 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.458072    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kxpz9_must-gather-m98rv_f92d07d7-15f2-4a71-b4ae-188ac7a7af73/copy/0.log"
Sep 17 18:42:39.458490 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.458465    2566 generic.go:356] "Generic (PLEG): container finished" podID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerID="0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e" exitCode=143
Sep 17 18:42:39.458545 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.458526    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:39.458582 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.458563    2566 scope.go:117] "RemoveContainer" containerID="0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e"
Sep 17 18:42:39.460659 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:42:39.460629    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-m98rv\" is forbidden: User \"system:node:ip-10-0-135-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kxpz9\": no relationship found between node 'ip-10-0-135-84.ec2.internal' and this object" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:39.467654 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.467626    2566 scope.go:117] "RemoveContainer" containerID="5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665"
Sep 17 18:42:39.469780 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:42:39.469750    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-m98rv\" is forbidden: User \"system:node:ip-10-0-135-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kxpz9\": no relationship found between node 'ip-10-0-135-84.ec2.internal' and this object" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" pod="openshift-must-gather-kxpz9/must-gather-m98rv"
Sep 17 18:42:39.480720 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.480698    2566 scope.go:117] "RemoveContainer" containerID="0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e"
Sep 17 18:42:39.481108 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:42:39.481087    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e\": container with ID starting with 0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e not found: ID does not exist" containerID="0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e"
Sep 17 18:42:39.481163 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.481117    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e"} err="failed to get container status \"0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e\": rpc error: code = NotFound desc = could not find container \"0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e\": container with ID starting with 0740cdb9c64634e5ac7a8b11f1acee81686bfa60498db1efffee82ed03da4c9e not found: ID does not exist"
Sep 17 18:42:39.481163 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.481136    2566 scope.go:117] "RemoveContainer" containerID="5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665"
Sep 17 18:42:39.481424 ip-10-0-135-84 kubenswrapper[2566]: E0917 18:42:39.481404    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665\": container with ID starting with 5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665 not found: ID does not exist" containerID="5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665"
Sep 17 18:42:39.481482 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.481430    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665"} err="failed to get container status \"5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665\": rpc error: code = NotFound desc = could not find container \"5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665\": container with ID starting with 5b753a86ed2db1745cf6d65dc07ce9524a9d70ef0c9ede736514668deb85c665 not found: ID does not exist"
Sep 17 18:42:39.511060 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.511023    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9k2\" (UniqueName: \"kubernetes.io/projected/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-kube-api-access-7t9k2\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:42:39.511060 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:39.511059    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f92d07d7-15f2-4a71-b4ae-188ac7a7af73-must-gather-output\") on node \"ip-10-0-135-84.ec2.internal\" DevicePath \"\""
Sep 17 18:42:40.507448 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:40.507411    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" path="/var/lib/kubelet/pods/f92d07d7-15f2-4a71-b4ae-188ac7a7af73/volumes"
Sep 17 18:42:44.018428 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.018387    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a/alertmanager/0.log"
Sep 17 18:42:44.041180 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.041151    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a/config-reloader/0.log"
Sep 17 18:42:44.070036 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.069989    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a/kube-rbac-proxy-web/0.log"
Sep 17 18:42:44.099708 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.099661    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a/kube-rbac-proxy/0.log"
Sep 17 18:42:44.121116 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.121087    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a/kube-rbac-proxy-metric/0.log"
Sep 17 18:42:44.146044 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.146019    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a/prom-label-proxy/0.log"
Sep 17 18:42:44.170458 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.170433    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70ff6b12-8cc9-4d90-a4d5-8f8d99522a4a/init-config-reloader/0.log"
Sep 17 18:42:44.372835 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.372810    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-t82j5_0bf563f2-03ff-4b66-982a-b40bf7d4501e/monitoring-plugin/0.log"
Sep 17 18:42:44.493827 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.493797    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ks9d5_4a5b31a0-b429-4ccc-8961-f246207f6791/node-exporter/0.log"
Sep 17 18:42:44.515582 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.515551    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ks9d5_4a5b31a0-b429-4ccc-8961-f246207f6791/kube-rbac-proxy/0.log"
Sep 17 18:42:44.536503 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.536481    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ks9d5_4a5b31a0-b429-4ccc-8961-f246207f6791/init-textfile/0.log"
Sep 17 18:42:44.774901 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.774873    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_00878956-76d9-480f-bb14-543a8c5bc19a/prometheus/0.log"
Sep 17 18:42:44.809103 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.809069    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_00878956-76d9-480f-bb14-543a8c5bc19a/config-reloader/0.log"
Sep 17 18:42:44.840592 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.840559    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_00878956-76d9-480f-bb14-543a8c5bc19a/thanos-sidecar/0.log"
Sep 17 18:42:44.870850 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.870781    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_00878956-76d9-480f-bb14-543a8c5bc19a/kube-rbac-proxy-web/0.log"
Sep 17 18:42:44.897889 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.897831    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_00878956-76d9-480f-bb14-543a8c5bc19a/kube-rbac-proxy/0.log"
Sep 17 18:42:44.922921 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.922889    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_00878956-76d9-480f-bb14-543a8c5bc19a/kube-rbac-proxy-thanos/0.log"
Sep 17 18:42:44.949124 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:44.949093    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_00878956-76d9-480f-bb14-543a8c5bc19a/init-config-reloader/0.log"
Sep 17 18:42:45.034381 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:45.034356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-9kd5n_5e0a6ea3-e191-42dd-ac7c-1e71f1b47d53/prometheus-operator-admission-webhook/0.log"
Sep 17 18:42:45.064893 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:45.064866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c87696bbc-wz6sm_1c2dda93-3811-41ed-8b56-ac82582ffdeb/telemeter-client/0.log"
Sep 17 18:42:45.089230 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:45.089196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c87696bbc-wz6sm_1c2dda93-3811-41ed-8b56-ac82582ffdeb/reload/0.log"
Sep 17 18:42:45.115957 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:45.115933    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c87696bbc-wz6sm_1c2dda93-3811-41ed-8b56-ac82582ffdeb/kube-rbac-proxy/0.log"
Sep 17 18:42:47.255143 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.255106    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"]
Sep 17 18:42:47.255554 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.255460    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerName="copy"
Sep 17 18:42:47.255554 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.255475    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerName="copy"
Sep 17 18:42:47.255554 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.255496    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerName="gather"
Sep 17 18:42:47.255554 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.255502    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerName="gather"
Sep 17 18:42:47.255685 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.255568    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerName="gather"
Sep 17 18:42:47.255685 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.255582    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92d07d7-15f2-4a71-b4ae-188ac7a7af73" containerName="copy"
Sep 17 18:42:47.257727 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.257708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.260305 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.260283    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6cb\"/\"kube-root-ca.crt\""
Sep 17 18:42:47.260438 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.260287    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6cb\"/\"openshift-service-ca.crt\""
Sep 17 18:42:47.261337 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.261311    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mv6cb\"/\"default-dockercfg-nvlfk\""
Sep 17 18:42:47.269645 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.269614    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"]
Sep 17 18:42:47.386043 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.386001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-podres\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.386234 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.386057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-proc\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.386234 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.386124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd957\" (UniqueName: \"kubernetes.io/projected/c871662f-d0a9-4199-bcc9-bbd62c7dca57-kube-api-access-rd957\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.386308 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.386242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-sys\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.386308 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.386258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-lib-modules\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.487706 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.487667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-podres\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.487930 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.487719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-proc\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.487930 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.487791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-proc\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.487930 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.487842    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-podres\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.487930 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.487838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd957\" (UniqueName: \"kubernetes.io/projected/c871662f-d0a9-4199-bcc9-bbd62c7dca57-kube-api-access-rd957\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.488160 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.487958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-sys\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.488160 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.487976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-lib-modules\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.488160 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.488079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-sys\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.488160 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.488081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c871662f-d0a9-4199-bcc9-bbd62c7dca57-lib-modules\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.497359 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.497322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd957\" (UniqueName: \"kubernetes.io/projected/c871662f-d0a9-4199-bcc9-bbd62c7dca57-kube-api-access-rd957\") pod \"perf-node-gather-daemonset-z9hx2\" (UID: \"c871662f-d0a9-4199-bcc9-bbd62c7dca57\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.568336 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.568301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:47.837518 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:47.837488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"]
Sep 17 18:42:47.840133 ip-10-0-135-84 kubenswrapper[2566]: W0917 18:42:47.840103    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc871662f_d0a9_4199_bcc9_bbd62c7dca57.slice/crio-412e6475fced194660d09ddc1021e679c56c438c0b9d1187e5b76397629d2e48 WatchSource:0}: Error finding container 412e6475fced194660d09ddc1021e679c56c438c0b9d1187e5b76397629d2e48: Status 404 returned error can't find the container with id 412e6475fced194660d09ddc1021e679c56c438c0b9d1187e5b76397629d2e48
Sep 17 18:42:48.499396 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:48.499354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2" event={"ID":"c871662f-d0a9-4199-bcc9-bbd62c7dca57","Type":"ContainerStarted","Data":"161239540af0f5fac27359d727542717bc5fbf784bed3e5583ccf33a86cbd6da"}
Sep 17 18:42:48.499806 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:48.499420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2" event={"ID":"c871662f-d0a9-4199-bcc9-bbd62c7dca57","Type":"ContainerStarted","Data":"412e6475fced194660d09ddc1021e679c56c438c0b9d1187e5b76397629d2e48"}
Sep 17 18:42:48.499806 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:48.499483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:48.518276 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:48.518155    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2" podStartSLOduration=1.5181340840000002 podStartE2EDuration="1.518134084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:48.514717434 +0000 UTC m=+4128.374308790" watchObservedRunningTime="2026-09-17 18:42:48.518134084 +0000 UTC m=+4128.377725443"
Sep 17 18:42:49.063647 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:49.063612    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dsw5m_57cb9384-de88-4d68-b887-c58189f35287/dns/0.log"
Sep 17 18:42:49.085241 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:49.085210    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dsw5m_57cb9384-de88-4d68-b887-c58189f35287/kube-rbac-proxy/0.log"
Sep 17 18:42:49.262119 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:49.262087    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npxnb_72e8ef0e-4a19-4a80-b7db-057ffafcfd50/dns-node-resolver/0.log"
Sep 17 18:42:49.787460 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:49.787419    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d4cc54796-5529z_1eef26e3-b4ac-42a5-a799-d87f15613b73/registry/0.log"
Sep 17 18:42:49.862128 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:49.862091    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h4cs9_14f32c3f-67dd-4ab0-9e2a-1ea09fae7f3d/node-ca/0.log"
Sep 17 18:42:50.888592 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:50.888563    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f955969b6-hrfzg_3417e40a-cdbd-4fd3-a68f-acca57237487/router/0.log"
Sep 17 18:42:51.412742 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:51.412711    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j4rzw_6a090127-3a2e-4f69-aed5-ceee9cc202d3/serve-healthcheck-canary/0.log"
Sep 17 18:42:51.930551 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:51.930521    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2gbv7_f3dca4f0-f8f3-4038-857c-a622206fa93f/kube-rbac-proxy/0.log"
Sep 17 18:42:51.956591 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:51.956563    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2gbv7_f3dca4f0-f8f3-4038-857c-a622206fa93f/exporter/0.log"
Sep 17 18:42:51.978163 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:51.978143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2gbv7_f3dca4f0-f8f3-4038-857c-a622206fa93f/extractor/0.log"
Sep 17 18:42:54.514396 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:54.514365    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-z9hx2"
Sep 17 18:42:55.000523 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:55.000491    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ff6d497d7-54kmf_da095f24-21d6-48c0-bebf-73425f088141/manager/0.log"
Sep 17 18:42:55.733512 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:55.733477    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7f654675b-5qgdq_ae7f6847-d7ff-4976-824a-f55af7f07f22/manager/0.log"
Sep 17 18:42:56.010147 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:56.010054    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-77658fbd58-w4mjv_572aec9b-9e51-4595-ad22-f5de1d8862aa/manager/0.log"
Sep 17 18:42:56.426482 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:56.426452    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f955969b6-hrfzg_3417e40a-cdbd-4fd3-a68f-acca57237487/router/0.log"
Sep 17 18:42:56.679265 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:56.679230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-wpj5d_6efe2f4c-63e7-4e75-9bdb-170767d4e22c/s3-init/0.log"
Sep 17 18:42:56.695378 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:56.695353    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-wpj5d_6efe2f4c-63e7-4e75-9bdb-170767d4e22c/download-hf-model/0.log"
Sep 17 18:42:56.723597 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:56.723558    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-wbpfh_75c9dd75-d94d-4263-aeaf-44d2579cb943/seaweedfs/0.log"
Sep 17 18:42:57.644566 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:42:57.644509    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f955969b6-hrfzg_3417e40a-cdbd-4fd3-a68f-acca57237487/router/0.log"
Sep 17 18:43:02.865163 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:02.865133    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dqjpg_717c1fea-058a-46b5-89c1-7b17e3b77db4/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:43:02.889958 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:02.889930    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dqjpg_717c1fea-058a-46b5-89c1-7b17e3b77db4/egress-router-binary-copy/0.log"
Sep 17 18:43:02.912878 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:02.912850    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dqjpg_717c1fea-058a-46b5-89c1-7b17e3b77db4/cni-plugins/0.log"
Sep 17 18:43:02.935336 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:02.935311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dqjpg_717c1fea-058a-46b5-89c1-7b17e3b77db4/bond-cni-plugin/0.log"
Sep 17 18:43:02.958099 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:02.958056    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dqjpg_717c1fea-058a-46b5-89c1-7b17e3b77db4/routeoverride-cni/0.log"
Sep 17 18:43:02.982502 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:02.982473    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dqjpg_717c1fea-058a-46b5-89c1-7b17e3b77db4/whereabouts-cni-bincopy/0.log"
Sep 17 18:43:03.004374 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:03.004338    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dqjpg_717c1fea-058a-46b5-89c1-7b17e3b77db4/whereabouts-cni/0.log"
Sep 17 18:43:03.464106 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:03.464020    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qx4hz_ba2ba32b-2d80-4def-8d77-2899b37d9f0b/kube-multus/0.log"
Sep 17 18:43:03.488455 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:03.488428    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4q9vq_3ed98aec-15bb-4997-89c2-6606d5c377be/network-metrics-daemon/0.log"
Sep 17 18:43:03.513830 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:03.513802    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4q9vq_3ed98aec-15bb-4997-89c2-6606d5c377be/kube-rbac-proxy/0.log"
Sep 17 18:43:04.443302 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.443270    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-controller/0.log"
Sep 17 18:43:04.464851 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.464822    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/1.log"
Sep 17 18:43:04.464851 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.464848    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovn-acl-logging/0.log"
Sep 17 18:43:04.485138 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.485105    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/kube-rbac-proxy-node/0.log"
Sep 17 18:43:04.507950 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.507924    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:43:04.530336 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.530308    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/northd/0.log"
Sep 17 18:43:04.565062 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.565034    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/nbdb/0.log"
Sep 17 18:43:04.591019 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.590988    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/sbdb/0.log"
Sep 17 18:43:04.653650 ip-10-0-135-84 kubenswrapper[2566]: I0917 18:43:04.653615    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmbx_be1e098d-32d9-456d-b2e8-cf856f7f7dd9/ovnkube-controller/0.log"